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Foreword 
By the Editors of the Series 


In the field of psychology we believe that the student 
ought to get the “feel” of experimentation by reading 
original source materials. In this way he can acquire a 
better understanding of the discipline by seeing scientific 
ideas grow and change. However, one of the main prob- 
lems in teaching is the limited availability of these sources, 
which communicate most effectively the personality of 
the author and the excitement of ongoing research. 

For these reasons we have decided to edit several 
books,* each devoted to a particular problem in psychol- 
` ogy. In every case we attempt to select problems that have 
been and are controversial—that have been and still are 
alive. We intend to present these problems as a set of 
selected original articles, arranged in historical order and 
in order of progress in the field. We believe that it is 
important for the student to see that theories and re- 
Searches build on what has gone before; that one study 
leads to another, that theory leads to research and then 
to revision of theory. We believe that telling the student 
this does not make the same kind of impression as letting 

'm see it happen in actuality. The idea is for the student 
to read and build ideas for himself. 

Suggestions for Use—These readings books can be used 
| by the student in either of two ways. They are organized 
$ So that, with the help of the instructor (or of the students 
if used in Seminars), a topic can be covered at length and 
in depth. This would necessitate lectures or discussions on 
articles not covered in the series to fill in the gaps. On the 
other „hand, each book taken alone will give a student a 
800d idea of the problem being covered and its historical 


$ Jackground as well as its present state and the direction 
| it seems to be taking. g 
ees 


z 
(Pub. note: a sub-series within the Insight Book Series. ) 
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Preface 


The field of programmed learning, or auto-instructional 
materials (erroneously called “teaching machines” ), is a 
rapidly developing, extremely exciting movement toward 
the further development of an instructional technology 
based, in part, on certain principles of psychology derived 
from laboratory studies of human and animal learning 
which serve to confirm, in part, the procedures and tech- 
niques used by nearly all truly great teachers of the past. 

Many professions other than psychology have a legiti- 
mate interest in instructional technology, e.g., specialists 
in the fields of audio-visual materials, curriculum develop- 
ment, engineering, systems development, teacher educa- 
tion, educational administration, and communications. 
This set of readings concerns itself with only a small part 
of the field of instructional technology, programmed 
learning, and it stresses that material of greatest interest 
to psychologists, particularly educational psychologists. 
Only in the first article, Finn’s “A New Theory of Instruc- 
tional Technology,” will the reader find the broad concern 
with this topic. Finn’s article represents, then, the broad 
context within which one might view some of the work of 
several professions, including psychology. 

_ Psychology’s concern with an instructional technology 
is rather recent, and it has been limited chiefly to evalu- 
ative studies of media. Its interest in programmed learn- 
ing and auto-instructional devices is far greater than its 
interest in other media, since two psychologists, B. F. 
Skinner and Sidney L. Pressey, have been the pioneers in 
the development of self-teaching devices and materials. 
Beginning with Skinner’s 1954 paper entitled “The Sci- 
ence of Learning and the Art of Teaching,” psychology’s 
claim to a contribution to developing an instructional 
technology gained considerable support. In a sense, Skin- 
ner proposed that laboratory studies of animal learning 
serve as a base for a science of teaching; however, earlier 
work with human learning is not to be considered irrele- 
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vant to the developments in programming. The work of * 


Ebbinghaus and his followers, of Thorndike and his stu- 
dents and of Pressey and his students inevitably bears a 
relationship to programmed learning. 

A casual perusal by a humanist of the rapidly growing 
literature on self-instructional devices and programmed 
material would probably give rise to either the fantasies 
of Alice in Wonderland and the Wizard of Oz or a vision 
of Brave New World, Walden Two and 1984. These 
views of the field, however, are quite inaccurate, as one 
can discover by critical reading of those articles prepared 
within a scholarly tradition rather than of those prepared 
within the frame-of-reference of technology and com- 
mercialism. Unfortunately, much of the activity in this 
area has been concerned with the development of ma- 
chines rather than with the development of materials for 
use with or without machines. Well over one hundred 
programs are being developed and, as a consequence, the 
respective roles of machines and Programs will become 
clearer within a few years. 


Until programmed material is available in reasonable 
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amounts, it is difficult to engage in research on the many 


variables which are of great interest to psychologists. Per- 
haps it might be said that one of the most exciting pos- 
sibilities before us is an opportunity for the first time to 
conduct research on the human learning of continuous 
discourse material with reasonable experimental control 


over the instructor variable. Machines are impartial, im- +° 


personal, and uniform. Teachers, fortunately, are personal 
and individual, but these desirable qualities frequently 
have been confounding variables in educational research 
on teaching methods and the parameters of learning 
under classroom or field conditions. The number of stud. 
ies on variables operating in programmed learning is in- 
creasing rapidly, and soon there will be data to support or 
refute the many claims which have been made. 

Our selection of the readings was guided by the several 
considerations set forth in the preceding paragraphs. The 
development and an overview of the field and some of the 
televant psychological theory are presented in Part I, the 
method—programming—is presented in Part II, and 
samples of research based on the methods of programming 
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are presented in Part III. The nature of teaching ma- 
chines or auto-instructional devices has been minimized 
deliberately in the selection of readings on the grounds 
that at the present stage of development of the field, the 
method for testing the most promising psychology theory 
for a science of teaching is the program which may or may 
not require a machine for presentation of the program 
content. Therefore, descriptions and functions of ma- 
chines will be found only as integral parts of articles, the 
primary concern of which is programming as a method or 
research on the parameters of learning from programmed 
material. 

Each new development in science tends to generate its 
own vocabulary, and this is particularly true of an inter- 
disciplinary subject like programmed learning. To help 
those readers who are unfamiliar with the particular 
terminolgy which has developed for this area, a glossary 
of teaching machine terms is given in Appendix A. 

For those readers who are interested in specific infor- 
mation on some of the existing machines and programs, 
Appendix B provides the names and addresses of several 
machine manufacturers and program distributors. Addi- 
tional information on most of the topics discussed in 
this book can be found in other publications, a list of 
which is also provided in Appendix B. 
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EDITOR’S NOTE 


The set of articles reprinted in this section 
was selected to define the problem and to 
present some of the issues and the back- 
ground for them. The educational problem 
which has given rise to the great interest in 
teaching machines and programming is ably 
presented by James Finn who has long been 
a leader in audio-visual education. The arti- 
cles by B. F. Skinner and James Holland de- 
scribe the theory and research in psychology 
upon which much of the recent work on 
programmed learning has been based. A 
somewhat different theoretical framework 
for programmed learning will be found in 
David Zeaman’s article. 

Professor Fry's article attempts to define 
an issue which perhaps has had more atten- 
tion than it warrants, viz., whether the mode 
of response is a critical variable in learning 
via programmed material. 

_A thorough, thoughtful and valuable re- 
view of the work on teaching machines and 
programming from a psychologist’s view is 
provided for the purpose of bridging theory, 
method, and research. 


Part I 


A New Theory for Instructional 
Technology* 


James D. Finn 
University of Southern California 


Technology! relates to education in at least three major 
ways. First, in a society in which science and technology 
are primary, such as America, the society requires that 
the educational system insure an adequate supply of 
scientists and associated technicians. This requirement 
sets a curriculum problem, an organization problem, and 
many other problems associated with the screening, selec- 
tion, and education of young people as potential addi- 
tions to the nation’s technical manpower. 

Second, as a society becomes more and more techno- 
logically oriented and controlled, the question of the 
general education of all citizens is raised. The survival 
and management of the whole society theoretically re- 
quire more general education in the sciences and tech- 


_ * This article was reprinted from Audio Visual Communica- 
tion Review, 1960, 8, 84-94. It is used here with the permission 
of the author. 

* For purposes of this paper, no distinction is made between 
“science” and “technology” or between “pure” and “applied 
science. The distinction between “basic” ed “applied” re- 
search is perhaps more meaningful, but these two activities are 
still regarded here as positions on a continuum. From a societal 
point of view, it matters little whether the society is hit with 
new ideas or new processes that stem from the ideas, or, as is 
more likely, a combination of the two. 
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nology for all. Pressures arise for more mathematics and 
science to be taught to the entire population. Again, 
curriculum problems, organization problems, and a host 
of nagging, persistent general education problems arise. 

Third, because of the tendency for technology to have 
no limits and constantly to extend into new areas, it is 
inevitable that, in an advanced technical society, tech- 
nology should begin to extend into the instructional 
process itself. As will be shown, this is particularly true 
when education has been, for a century or more, one 
of the areas of American society which has walled itself 
off from technological advances and, consequently, has 
created a technological vacuum. That vacuum is now 
rapidly being filled. 

It is with this third relationship—the application of 
technology to the educational, or to be more precise, to 
the instructional process—that the balance of this paper 
is concerned. The three relationships just discussed— 
development of technicians, general education in tech- 
nology, and the application of technology to the instruc- 
tional process—cannot eventually be completely separated. 
However, the third relationship is sufficiently different 
to merit thorough analysis. . 

Again for purposes of this paper, a finer distinction will 
now be drawn. Within the educational process itself, 
there are three general areas in which technology can or 
is being applied. These are: (1) general administration, 
(2) testing, and (3) instruction. The uses of technical 
management systems, modern equipment, etc., repre 
sent the fairly obvious applications to the field of general 
administration. While it probably can easily be show? 
that this area of management is, taken as a whole, tw 
or three decades behind its counterpart in industry, the 
problems associated with technology and school man- 
agement are not as difficult as some of the others an 
will not, at this time, be considered. This is not to say 
that administration is not highly interrelated with the 
other two areas of testing and instruction, but the pos 
tion can be taken that the problems arising from the 
latter two should guide the technical solutions in adminis- 
tration, not the other way around. 

The second category, testing, is in many respects the 
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most highly developed technology at present existing in 
American education. This is true both from a machine 
and from a systems standpoint. Further, the close rela- 
tionships between psychological, achievement, and other 
types of testing and the instructional process are so well 
defined as to need no comment. However, with some 

+ exceptions, the technology of the instructional process 
can, for purposes of analysis, be isolated from testing. 
This arbitrary decision is made here in order to further 
the remainder of the discussion. We are left, then, with 
the instructional process by itself and can now turn to 
the impact of technology upon that process. 


DEVELOPMENT OF INSTRUCTIONAL TECHNOLOGY 


The development of a technology of the instructional 
Process is relatively new. In the pre-industrial phases of 
both education and industry, while industry was prin- 
cipally at the handwork, artisan level, the instructional 
process relied upon such devices as the slate, the horn- 
book, the blackboard, chalk, and limited single text- 
books with few illustrations. Although attention will 
be confined principally to some of the symbols of tech- 
nology—equipment and machines—it should be em- 
phasized that other factors were in the same state. In 
the area of organization, for example, it is well known 
_ that the graded school is a late development. 

_ We note the famous changes jn industry at the be- 
ginning of the 19th century—the invention of a group 
of related machines for power, spinning, weaving, etc.— 
which made possible the factory system. Based on the 
work of Toynbee the elder, the term “Industrial Revolu- 
tion” is applied to this period. In education, however, 
the same revolution did not occur and instructional 
technology (with some exceptions here and there) re- 
mained at the pre-industrial level. During the last quar- 
ter of the century, there was some indication 0} incipient 
change. Oliver (1956) notes that an exhibit from an 
. American school “with maps, charts, textbooks _and 
fy other equipment” won admiration at the International 
| Exposition in Vienna in 1873. He adds that the American 


3 Now called the First Industrial Revolution. 
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school display at Paris in 1878 was even more outstand- 
ing. Significantly, Oliver's last mention of international 
attention to American instructional technology was at 
Melbourne in 1880. 

By 1900 industry had, in contrast, established factories 
and was moving into assembly line operations; had be- 
gun to apply, in a crude way, research and development 
concepts; had introduced the beginnings of modern man- 
agement; and had developed a sophisticated financial 
system. Developments had reached such a state that the 
perceptive Henry Adams, true prophet and seer of tech- 
nology, foretold the coming of the Age of the Atom and 
the problems it would bring. In the years to 1950, aided 
by the acceleration of two wars, technology burgeoned 
and developed, piled machine upon machine, system upon 
system, added fantastically to its power resources, and 
invented the method of invention. Technology trans- 
formed American society, philosophy, and art. 

During the period from 1900 to 1950, however, tech- 
nology only washed lightly upon the shores of instruc- 
tion. In this same time span, when high speed printing 
techniques, radio, sound motion pictures, television, and 
other pieces of communication technology were invented, 
developed, and exploited, American education failed to 
apply these devices in quantity to the instructional proc- 
ess and, of course, failed to develop the appropriate 
technological systems necessary for this application. There 
were always rumblings, to be sure, as evidenced by the 
statement attributed to Edison in 1916 that the moticn 
picture would replace the teacher. However, looked at 
from the vantage point of 1960, laboratories, project 
methods, libraries, and minute arrangements for audio- 
visual materials—the provisions to 1950 constituted 
what was still a pre-industrial technology for instruction. 

By 1950, industry had entered what may be called the 


pre-automation period, which heralded the beginning 


of a movement toward true automation. Much of the 
basic work on*com 


puters and other control mechanisms 
had been done. At this time also, mechanical systems 


of transporting work goods between machines had inte- 
grated some production lines into a system called “De- 
troit automation.” The period of true automation was 
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“dawning, and I have arbitrarily set 1955 as the date at 
which this electronic-mechanical-systems analysis proce- 
dure began to be significantly applied. It is no accident 
that man began to push toward space at about the same - 

` time. 

By 1950, American education had the potentiality (to 
carry on with the analogy) of a mass production tech- 
nology. The hardware—projectors, recorders, television 
—and the materials were present. The systems concepts 
—saturation with audiovisual materials at the point of 
an instructional problem, for example, a concept derived 
from the military experience of World War I]—were 
developed and known in a few audiovisual circles. A 
certain amount of incentive and public acceptance, also 
derived from World War II experience, could be drawn 
upon. The cake of custom, however, proved to be too 
tough. The mass production stage, at least 100 years 
behind industry, was not entered except here and there 
on isolated little islands. 

It was at this point, approximately in 1955, that the 
god came out of the machine in the form of the Ford 
Foundation. Dr. Alvin C. Eurich (and a little later, his 
associate, Dr. Alexander J. Stoddard) gave the instruc- 
tional processes of American education a sharp push into 
mass production technology. The time was ripe. There 
was a shortage of teachers; education and educationists 
were under fire from all sides; Neo-Technocracy was turn- 
ing its attention to education; the race with Russia was 
underway; the natives were restless indeed. 

There are several interesting facets to this shove into 
mass instructional technology. First, television, both 
closed-circuit and broadcast, was chosen as the prime hard- 
ware. This was due, I think, to the unconsciously as- 
sumed basic concept of mass production, usually stated 
in terms of a shortage of teachers, large classes, and 
quality instruction. Second, the Ford impetus made pro- 
vision, for the first time, for a technology of systems to 
go along with the hardware. Stoddard’s sown work in 
designing the school of tomorrow (1957) and the work 
of Trump and his associates in drawing the instructional 
images of the future for the American secondary school 
(1959) are, essentially, proposed systems. 
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Third, the Ford group made use of technology in 
creating the impetus in the first place. The technology 
of social psychology and public relations was drawn upon; 
educational decision-makers were the prime targets, and 
the means used, including high-priced public relations 
counsel, were the best available. This accounts for the 
fact that the teaching profession in general and audio- 
visual specialists in particular—the latter are really the 
only technologists of the profession—were for the most 
part left out of this move into technology. 

Instead, the Foundation went to administrators and 
board members and influential opinion makers of all — 
types. Organizations were approached or created: the » 
National Educational Television and Radio Center (now, — 
I believe, under the administration of the Learning Re- | 
sources Institute), the revitalized National Association of 
Educational Broadcasters, the Educational Facilities Lab- 
oratories, Inc., and others. As a sign of this policy to 
short-circuit the profession, it is significant that, except 
for a small grant to promote a seminar or two, the 
Department of Audiovisual Instruction of the National » 
Education Association was left out of the largesse. 

The Ford people assumed, I think, that if the cake of 
custom was to be broken the Priesthood was not the 
agency to break it. This may mean, in part and as far 
as audiovisual and curriculum people are concerned, that 
to an outsider their technological orientation is at the 
level of the water wheel and the handloom, not the 4 
computer and the rocket. It is something to think about. 
On the other hand, it may also mean in part that the 
Ford Foundation is merely another instrument of Neo- 
Technocracy and that these actions forecast even more 


loss of control of education by the existing pre-techno- 
logical profession. 


IMPACT OF PRESENT TRENDS 


With this background of development painted in, we 
can now come to gtips with the impact of the present 
technology. If we consider education as a societal uni- 
verse, what happens when we introduce energy into that 


universe suddenly and on a large scale? That is ap- 
parently what is happening. 
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Two general effects can be predicted by basing our 
thinking on two analogous concepts. If the concept o 
entropy holds good in this connectiom, the educationa 
system will become more highly organized and less ran- 
dom in nature (negative entropy will increase); if the 
laws of social change first stated by William G. Sumner 
and Wiliam F. Ogbum apply, the introduction of a 
system of technology into the culture will cause dis- 
tortions, stresses, and non-predictable effects to increase 
until a crisis is reached and the culture becomes much 
different. Two historical events serve as examples, one 
taken from Mumford (1955) (the introduction of the 
clock to regulate the habits of Monks) and one from 
Muller (the invention of writing in Egypt for the pur- 
pose of keeping accounts) (1959). Both changed the 
world several times over. The effects of neither were, of 
course, predictable. 

hat, then, are the present trends? If you don’t count 
the continuous, grinding, sweating struggle carried on by 
audiovisual specialists, dealers, manufacturers, and pro- 
ducers to introduce, one item at a time, film libraries, 
Projectors, recorders, etc., into the educational system in 
the slow process of conversion from a pre-industrial 
technology, two major trends can be identified. These 
two trends, at the moment, lead in opposite directions. 

The first is the trend toward a mass instructional 
technology and is governed by machines and systems 
suitable for that purpose. Foremost, of course, is tele- 
vision, of which there are four instructional types: (1) 
broadcast on an educational channel, (2) „broadcast 
on a commercial channel, (3) closed-circuit of the 
Hagerstown-Penn State type in which live instructors 
are used either to supplement instruction or to provide 
direct instruction exclusive of classroom teachers, and 
(4) the Compton type in which filmed lectures are dis- 
tributed via the closed-circuit medium as replacement for 
classroom teachers. In all cases, the desire is to reach 
more students with fewer teachers or to obtain “quality 
instruction. : 

Mass instruction technology in another form includes, 
of course, the massed film systems. The prime example 
is the EBF series in physics and chemistry, amounting 
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to over 300 half-hour motion pictures intended to be 
used where there are no science teachers, where there are 
teachers not considered qualified, as audiovisual aids to 
more qualified teachers, and with very large groups of 
students. These films, of course, may theoretically be 
used over television. The exploitation of the overhead 
transparency projector is another example of mass instruc- 
tional technology as shown in the Newton, Massachusetts 
experiment in English and grammar and composition. 
In opposition to this trend of mass instruction is a 
growing technology for individual instruction. This trend 
is the audiovisual wave of the future (for the moment). 
The most dramatic development here is that class of $ 
systems and instruments known as teaching machines- | 
1 
| 


Actually, I would class all teaching equipment designed 
for individual or near-individual operation as being in the 
category of teaching machines.? At present, then, there 
are approximately five types, listed here on an ascending 
scale of sophistication: (1) individual reading papers 
and similar devices, (2) individual viewing and listening 
equipment for existing slides, filmstrips, motion picturcs, 
and recordings, (3) language laboratories of all types» 
(4) specifically programmed printed materials such as 
scrambled textbooks and the Lumsdaine notebook, an 
(5) true teaching machines of the Skinner or Pressey type 
containing carefully worked out verbal or pictorial pro” | 
grams with various ingenious mechanical or electronic f 
arrangements to test student reaction and inform him 
of his progress, errors, etc. | 
f 


EXTRAPOLATION I 


These two major trends toward technologies of mass 
and of individual instruction are not science fiction; they 
are with us now. Assessment of this impact is our most 
pressing and difficult problem. It does not take much of — 
a crystal ball to see that a combination of these two 
technologies i> the next immediate step. For example, 
let us take Maurice Mitchell’s new series for his chemistry l 
course on 150 plus films. Let us further assume that Mr. 


r ne 
a Preferred terms in some quarters are “sel f-instructional and 
auto-instructional” devices. 
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™ Mitchell makes a deal with the Rheem Califone Corpo- 
ration to program their teaching machine with material 
to go along with the chemistry films. (Mr. Eurich might 
have to put up the money for the initial effort.) Let us 
continue by saying we will project the films hourly on the 
Compton College closed-circut television film distribu- 
tion system and place the machines in proper cubby- 
holes for Compton College students to work at specified 
times. We could get, almost immediately, to what can 
be called total educational automation. 

Or, in another context, let us suppose that the new 
Ford creation, the Learning Resources. Institute, initiates 
another Continental Classroom in mathematics over na- 
tional broadcast television; and that the new audiovisual 
teaching machines now being developed are Be peer 
to relate to this new series; and that these machines are 
placed by the foundation in every public library in 
America; and that, at the conclusion of the instructional 
period, students report to designated centers where the 
Educational Testing Service examines them and certifies 
them to the colleges of their choice. Think, for a mo- 
ment, about that one. It is now possible not only to 
eliminate the teachers but the school system.* 

These may be considered extreme applications, but 
such applications, at least experimentally, are inevitable. 
In fact, one such study, testing the combination of these 


s two technologies, is now underway. The point is that 


both the mass instruction systems and the technology 
of individual instruction—teaching machines—are getting 
terrific momentum. These technologies are going to hit 
education with a million-pound thrust. What will be the 
effect upon our educational society? What is the role of 
the teacher, the audiovisual specialist, the curriculum 
director? Quo vadis the curriculum itself? This last ques- 
tion is particularly pertinent if the medium or image 
governs the message, as McLuhan maintains. 


“Time magazine had some fun lifting this*sentence out of 
context. See Time, March 14,. 1959, p. 68. Time also said: 
Finn offered one caveat: this awesome equipment must_not 
fall into the hands of any one private institution, e.g., the Ford 
Foundation. Said he, “The American people don’t elect repre- 
sentatives to the Ford Foundation.’ ” 
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EXTRAPOLATION II 


Technology never sits still. The extrapolations given 
in the last section are immediate. What of those that 
can be made for a time slightly farther into the future? I 
see several developments. First, it is reasonable to expect 
that programming for teaching machines will move from 
the verbal-Socratic-Skinner type to the audiovisual-branch- 
ing type. That is, the machines will present conceptual 
content based upon student pretests, using films, slides, 
filmstrips, tapes, and/or videotape as the medium. The — 
Presentation sequences will be longer and the student £ 
will be given an Opportunity to select additional sequences 
for further explanation if the machine, through testing, 
informs him that he needs it. Records, of course, will be 
maintained instantaneously by miniaturized computers. 

Secondly, the work in England of Ashby, Pask, and 
others with teaching machines based on the design con- 
cepts of biological computers will affect the technology of 
the present teaching machines much as transactional psy- ' 
chology affects stimulus-response psychology. Pask has 
already produced a machine which, in training key punch 
operators, actually senses the characteristics of the stu- 
dent as he works and automatically adjusts the program 
of the machine to the student’s individual needs. This is | 
a pure transaction, and Pask maintains he can develop i 
such a machine to teach decision-making with present ® 
hardware and know-how (1958). | 

A second development of the future begins next fall 
with the Airborne TV experiment of the Ford Founda- 
tion over a city in Indiana. Circling overhead one or 
more airplanes carrying multiple television transmitters f 
can broadcast signals over much greater distance than © 
conventional tower transmission. The programs will go 
into several states and, because of the location, into both | 
urban and rural areas and into innumerable heretofore 
independent sehool districts. Effects of this on local j 
control of curricula, curriculum planning procedures, | 
Images of master teachers, patterns of school organiza- + 
tion, etc., have yet to be speculated about, let alone ex- 


plored. Nothing technical would prevent, by the way, 
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“a national network of such aircraft as an addition to our 


other national means of transmission. : 

A third predicted development is based on information 
that it is technically possible, although not yet_econom- 
ically feasible, to produce a motion picture film with 
characteristics similar to the Polaroid film that is instan- 
taneously developable. What this would do to videotape, 
to production techniques, and to the possibilities of 
local production of materials is of great interest to the 
audiovisual field.® 


TEACHER CONTROL 


The fourth development that can be anticipated is 
the most spectacular. Essentially, it is based on a sys- 
tems concept. It is theoretically possible now to design 
an automatic classroom under the control of the teacher. 
Most of the elements are present or can be designed. 
Such a classroom would have total light and air control, 
automatic projection and television systems, technical 
provision for the best possible dicussion environment, dis- 
play situations, etc., which could be changed at will. By 
planned programming, the classroom could be made to 
function for major presentations, small group discus- 
sions, individual work at teaching machines, creative 
periods, and the like. As we said, all of this could be 
under the control of the teacher. The classroom then 
would become the teaching machine. Adrian TerLouw at 
Eastman Kodak has done some work in this area and it 
is, of course, implied in the provocative article by Simon 
Ramo which is so often cited in discussions of teaching 
machines (1957). An acquaintance of mine refers to 
this concept as the “mad scientist” classroom. 


IMPLICATIONS 


To assess the implications of this current and predicted 
impact of technology on the instructionalptocess 1s not 
easy. As with any such speculation, it 1s also hazardous. 
It is, however, a job we must attempt. 


©The new thermo-plastic method of image-recording devel- 
oped by General Electric can do much the same thing. 
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We have to start, I think, with the proposition that ~ 


our educational society is in the position of a backward 
or underdeveloped culture suddenly assailed by the 20th- 
century engineer. In addition, reactions in some profes- 
sional circles to the advent of television are similar to 
those factory workers of the 19th century who attempted 
to destroy the machines that were taking over their jobs. 
In the long run, the machines remained; instructional 
technology is, no doubt, here to stay. Our problem be- 
comes one, not so much of how to live with it on some 
kind of feather-bedding basis as how to control it so 
that the proper objectives of education may be served 
and the human being remain central in the process. 
Certain things are obvious. I think the concept ©! 
negative entropy® will hold. The thrust and energy of 
technology will force a greater organization upon us at 
every point at which it is applied to instruction. Such 
an arrangement as the Airborne experiment will require 
a tighter and different organization in those five or SIX 
states and in those many school districts of the Midwest. 
The closed-circuit experiment begun a year ago in Ana- 
heim, California, is already, it seems to me, making 
requirements for organization—of scope and sequence 
in the curriculum, of the district itself, and of the nature 
of the television presentations—the rigor of which has 
never been felt before in Anaheim. 
, Programming, in its machine, computer, and automa- 
tion sense, is a matter of extreme organization—the piling 
up of energy.7 Most of the devices and systems of the new 
technology require, one way or another, this type ° 
programming—television, teaching machines, mass films, 
language laboratories. This means that programmers are 
needed. Who is to do this? This is essentially, in the 
old sense, both a curriculum and an audiovisual problem. 


“In all the tests I have been able to think of, this concept 
holds. I am almost convinced that this idea can achieve some 
status as, forgive'me, “Finn’s law.” 

The role of energy in relation to AV communication theory 
needs much study. For example, the energy involved in the 
negative entropy concept should be examined in much the 


ae eg Cottrell studied energy in general in Energy 
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It is also a social problem. The heartland is programming. 
He who controls the programming heartland controls the 
educational system. Will it be a foundation, a committee 
or scientists, textbook publishers and film producers, the 
NEA, the school superintendent, the board of education, 
the students, or the general public? 

Too, how many of us will go overboard and sink with 
the old concepts that will be absorbed or outmoded and 
tossed to the sharks by the new technology? Take the 
concept of instructional materials which some curriculum 
specialists love so well and which caused quite a con- 
troversy in DAVI circles a few years ago. The fight was- 
futile, as is the love. The concept of programming and 
the systems and systems analysis it implies completely 
absorbs the idea of materials. Instructional materials 
becomes an outmoded, atomistic, pre-technological con- 
cept useful mainly to the historians of education.8 

The concept of audiovisual education may accompany 
instructional materials down the drain, or it may not, 
depending on whether or not it can be redefined ac- 
ceptably.? The Skinner and Pressey type teaching ma- 
chines and their descendants are, for example, primarily 
verbal devices, and yet their management and program- 
ming as technical electronic devices belong somewhere. 
At a practical level, within a school situation, someone 
is going to have to worty about them. It is my conten- 
tion that the audiovisual field is in the easiest position 


to help integrate these mechanisms properly into the 
instructional process. They are not primarily audio- 
visual; they are primarily technological. The AV field, I 
think, must now suddenly grow up. We, the audiovisual 
specialists, are, of all educational personnel, the closest 
to technology now; we have, I think, to become specialists 
in learning technology—and that’s how I would redefine 
audiovisual education. 
What of the curriculum people? Their work is just 
beginning. It can become infinitely more rigorous and 
o 
s McLuhan maintains the same thing about audiovisual 
aids (1959). a 
° Redefinition (and terminology) of the audiovisual field 
are contemplated as subject of a future special supplement to 
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at the same time more satisfying. I suspect some of 
the emotional Rousseauians will have to get out of the 
curriculum business. The rest can step in and face this 
problem of programming a learning technology in human 
terms and find great excitement and reward. ‘The profes- 
sion as a whole must be made to sense this powerful 
movement to instructional technology and be made 
teady to seize the great opportunity it offers to make 
all teachers highly professional. 


THE PHILOSOPHY OF ADVENTURE 


In closing, I should like to quote Whitehead, who 
said that it is the business of the future to be dangerous. 
Technology certainly has made this aphorism into the 
outstanding fact of our time. Technology is now making 
the future of instruction capricious and hazardous. But 
in doing so it has presented us with more opportunity 
and more choices than ever before. If the future is an 
adventure, it is an adventure because of technology. The 
cost of civilization is the fact that we can make wrong 
choices because of the alternatives technology presents. 
The reward of civilization is the freedom provided by 
technology and the opportunity to make the right choices. 
This cost and this reward we now face with the tech- 
nology of the instructional process. We must look for- 
ward to the adventure and not present what Herbert 
Muller noted as “the curious spectacle of civilized man 
forever marching with his face turned backward—as no 
doubt the cave man looked back to the good old days 
when men were free to roam instead of being stuck in 4 
damn hole in the ground” (p. 65). The “good old days” 


are gone; approached with intelligence and zest, the days 
of the future will be better. 
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The Science of Learning and the 
Art of Teaching* 


> 
B. F. SKINNER 
Harvard University 
Some promising advances have recently been made in- 
the field of learning. Special techniques have been de- 
signed to arrange what are called “contingencies of 
reinforcement” —the relations which prevail between be- 
havior on the one hand and the consequences of that 
behavior on the other—with the result that a much more | 
effective control of behavior has been achieved. It has 
long been argued that an organism learns mainly by pro- | 
ducing changes in its environment, but it is only recently 
that these changes have been carefully manipulated. In } 
traditional devices for the study of learning—in the serial ~ 
maze, for example, or in the T-maze, the problem box, 
or the familiar discrimination apparatus—the effects pro- 
duced by the organism’s behavior are left to many fluc- 
tuating circumstances. There is many a slip between 
the turn-to-the-right and the food-cup at the end of the | 
alley. It is not surprising that techniques of this sort — 
have yielded only very rough data from which the uni- 
formities demanded by an experimental science can be 


‘ 
* Paper presented at a conference on Current Trends in 
Psychology and the Behavioral Sciences at the University of 
Pittsburgh, March 12, 1954. This article is reprinted from 
the Harvard Educational Review, 1954, 24, 86-97, and is used 
here with the permission of the author and the editors. | 
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extracted only by averaging many cases. In none of this 
work has the behavior of the individual organism been 
predicted in more than a statistical sense. The learning 
processes which are the presumed object of such research 
are reached only through a series of inferences. Current 
preoccupation with deductive systems reflects this state 
of the science. i 

Recent improvements in the conditions which control 
behavior in the field -of learning are of two principa 
sorts. The Law of Effect has been taken seriously; we 
have made sure that effects do occut and that they occur 
under conditions which are optimal for producing the 
changes called learning. Once we have arranged the 
particular type of consequence called a reinforcement, - 
our techniques permit us to shape up the behavior of an 
organism almost at will. It has become a routine exercise 
to demonstrate this in classes in elementary psychology 
by conditioning such an organism as a pigeon. Simply by 
presenting food to a hungry pigeon at the right time, it 
is possible to shape up three or four well-defined responses 
ma single demonstration period—such responses as 
turning around, pacing the floor in the pattern of a figure- 
8, standing still in a corner of the demonstration appara- 
tus, stretching the neck, or stamping the foot. Extremely 
complex performances may be reached through succes: 
sive stages in the shaping process, the contingencies o 
reinforcement being changed progressively in the direc- 
tion of the required behavior. The results are often quite 
dramatic. In such a demonstration one can see learning 
take place. A significant change in behavior is often ob- 
vious as the result of a single reinforcement. 3 

A second important advance in technique permits us 
to maintain behavior in given states of strength for long 
periods of time. Reinforcements continue to be impor- 
tant, of course, long after an organism has learned how 
to do something, long after it has acquired behavior. 
They are necessary to maintain the behavior in strength. 
; he effect of various schedules of 


OF special interest is t 
intermittent reinforcement. Charles B. Ferster and the 
sive report of a 


author are currently preparing an exten 
five-year research program, sponsored by the Office of 
Naval Research, in which most of the important types of 
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schedules have been investigated and in which the effects 
of schedules in general have been reduced to a few 
principles. On the theoretical side we now have a fairly 
good idea of why a given schedule produces its appropriate 
performance. On the practical side we have learned how 
to maintain any given level of activity for daily periods 
limited only by the physical exhaustion of the organism 
and from day to day without substantial change through 
out its life. Many of these effects would be traditionally 
assigned to the field of motivation, although the prt 
cipal operation is simply the arrangement of contin- 
gencies of reinforcement. J 

These new methods of shaping behavior and of main” 
taining it in strength are a great improvement over the 
traditional practices of professional animal trainers, an 
it is not surprising that our laboratory results are already 
being applied to the production of pérforming anima’s 
for commercial purposes. In a more academic environ- 


ment they have been used for demonstration purposes — 


uch. 


which extend far beyond an interest in learning aS 5 Jex 
P. 


For example, it is not too difficult to arrange the com 
contingencies which produce many types of socia”, 
havior. Competition is exemplified by two pigeons playing 
a modified game of ping-pong. The pigeons drive the f 
back and forth across a small table by pecking at ‘4 
When the ball gets by one pigeon, the other is reinforce’ 
The task of constructing such a “social relation. 
probably completely out of reach of the traditional anim 
trainer. It requires a carefully designed program se 
gradually changing contingencies and the skillful" 
of schedules to maintain the behavior in strength. et, 
pigeon is separately prepared for its part in the ay 
performance, and the “social relation” is then arbitra is 
constructed. The sequence of events leading up to the 
stable state are excellent material for the study ot is 
factors important in nonsynthetic social behavior. tine 
instructive to. consider how a similar series of con 


. some 
1 The reader may wish to review Dr. Skinner's article, $ a 
Contributions of an Experimental Analysis of Behavidgs3, 


Psychology as a Whole,” The American Psychologist, 1 
8, 69-78. Ed. 
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gencies could arise in “he ‘case “of the Hamat ‘organism 
through the evolution of cultural patterns. E 
Cooperation can also be set up, perhaps more easily 
than competition. We have trained two pigeons to CO- 
ordinate their behavior in a cooperative endeavor with a 
precision which equals that of the most skillful human 
dancers. In a more serious vein these techniques have 
permitted us to explore the complexities of the individual 
organism and to analyze some of the serial or coordinate 
behaviors involved in attention, problem solving, various 
types of self-control, and the subsidiary systems of Te 
sponses within a single organism called “personalities. 
Some of these are exemplified in what we call multiple 
schedules of reinforcement. In general a given schedule 
has an effect upon the rate at which a response is emitted. 
Changes in the rate from moment to moment show 
a pattern typical of the schedule. The pattem may be as 
simple as a constant rate of responding at a given value, 
it may be a gradually accelerating rate between certain 
extremes, it may be an abrupt change from not respond- 
ing at all to a given stable high rate, and so on. It has 
been shown that the performance characteristic of a 
given schedule can be brought under the control of a 
particular stimulus and that different performances can 
be brought under the control of different stimuli in 
the same organism. At a recent meeting of the American 
Psychological Association, Dr. Ferster and the author 
demonstrated a pigeon whose behavior showed the pattern 
typical of “fixed-interval” reinforcement in the presence 
of one stimulus and, alternately, the pattern typical of 
the very different schedule called “fixed ratio” in the 
presence of a second stimulus. In the laboratory we 
have been able to obtain performances appropriate to 
nine different schedules in the presence of appropriate 
stimuli in random alternation. When Stimulus | is pres- 
ent, the pigeon executes the performance appropriate to 
Schedule 1. When Stimulus 2 is present, the pigeon 
executes the performance appropriate to Sthedule 2. And 
so on. This result is important because it makes the extra- 
polation of our laboratory results to daily life much more 
plausible. We are all constantly shifting from schedule 
to schedule as our immediate environment changes, but 
Sly 4 AC? 7 
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the dynamics of the control exercised by reinforcement 
remain essentially unchanged. 

It is also possible to construct very complex sequences 
of schedules. It is not easy to describe these in a few 
words, but two or three examples may be mentioned. In 
one experiment the pigeon generates a performance 
appropriate to Schedule A where the reinforcement 18 
simply the production of the stimulus characteristic of 
Schedule B, to which the pigeon then responds appro- 
priately. Under a third stimulus, the bird yields a perform- 
ance appropriate to Schedule C where the reinforcement 
in this case is simply the production of the stimulus 
characteristic of Schedule D, to which the bird then 
responds appropriately. In a special case, first investigated 
by L. B. Wyckoff, Jr., the organism responds to one 
stimulus where the reinforcement consists of the clarift- 
cation of the stimulus controlling another response. The 
first response becomes, so to speak, an objective form 
of “paying attention” to the second stimulus. In one 
important version of this experiment, as yet unpublished, 
we could say that the pigeon is telling us whether it 1 

paying attention” to the shape of a spot of light or tO its 
color. 

„One of the most dramatic applications of these tech- 
niques has recently been made in the Harvard Psycho- 
logical Laboratories by Floyd Ratliff and Donald Ss. 
Blough, who have skillfully used multiple and seria 
schedules of reinforcement to study complex perceptua 
processes in the infrahuman organism. They have achieve! 
a sort of psycho-physics without verbal instruction. In 4 
recent experiment by Blough, for example, a pigeo” 
draws a detailed dark-adaptation curve showing the char- 
acteristic breaks of rod and cone vision. The curve }§ 
recorded continuously in a single experimental period an! 
is quite comparable with the curves of human subjects- 
The pigeon behaves in a way which, in the human case, 
we would not hesitate to describe by saying that it adjusts 
a very faint patch of light until it can just be seen. 

In all this work, the species of the organism has made 
surprisingly little difference. It is true that the organisms 
studied have all been vertebrates, but they still cover a 
wide range. Comparable results have been obtained with 
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pigeons, rats, dogs, monkeys, human children, and most 
recently, by the author in collaboration with Ogden R. 
Lindsley, human psychotic subjects. In spite of great 
phylogenetic differences, all these organisms show amaz- 
ingly similar properties of the learning process. It should 
be emphasized that this has been achieved by analyzing, 
-the effects of reinforcement and by designing techniques: 
which manipulate reinforcement with considerable preci- 
sion. Only in this way can the behavior of the individual 
organism be brought under such precise control. It is also 
important to note that through a gradual advance to 
complex interrelations among responses, the same degree 
a of rigor is being extended to behavior which would usually 
© be assigned to such fields as perception, thinking, and 
personality dynamics. 

From this exciting prospect of an advancing science of 
learning, it is a great shock to turn to that branch of 
technology which is most directly concerned with the 

| learning process—education. Let us consider, for ex- 
ample, the teaching of arithmetic in the lower grades. 
jì The school is concerned with imparting to the child a 
large number of responses of a special sort. The responses 
| are all verbal. They consist of speaking and writing cer- 
tain words, figures, and signs which, to put it roughly, 
| refer to numbers and to arithmetic operations. The first 
| task is to shape up these responses—to get the child to 
s pronounce and to write responses correctly, but the prin- 
WH cipal task is to bring this behavior under many sorts of 
© stimulus control. This is what happens when the child 
learns to count, to recite tables, to count while ticking 
off the items in an assemblage of objects, to respond to 
spoken or written numbers by saying “odd,” “even,” 
W “prime,” and so on. Over and above this elaborate reper- 
toire of numerical behavior, most of which is often dis- 
missed as the product of rote learning, the teaching of 
arithmetic looks forward to those complex serial arrange- 
ments of responses involved in original mathematical 
thinking. The child must acquire responses ‘Of transposing, 
clearing fractions, and so on, which modify the order or 
ìb pattern of the original material so that the response called 
a solution is eventually made possible. 
Now, how is this extremely complicated verbal reper- 


al a7 al 


24 SCIENCE OF LEARNING AND Arr OF TEACHING 


toire set up? In the first place, what reinforcements are 
used? Fifty years ago the answer would have been clear. 
At that time educational control was still frankly aver- 
sive. The child read numbers, copied numbers, memorized 
tables, and performed operations upon numbers to escape 
the threat of the birch rod or cane. Some positive rein- 
forcements were perhaps eventually derived from the 
increased efficiency of the child in the field of arithmetic 
and in rare cases some automatic reinforcement may have 
resulted from the sheer manipulation of the medium— 
from the solution of problems or the discovery of the 
intricacies of the number system. But for the immediate 
purposes of education the child acted to avoid or escape 
punishment. It was part of the reform movement known 
as progressive education to make the positive conse- 
quences more immediately effective, but any one who 
visits the lower grades of the average school today will 
observe that a change has been made, not from aversive 
to positive control, but from one form of aversive stimu- 
lation to another. The child at his desk, filling in his 
workbook, is behaving primarily to escape from the threat 
of a series of minor aversive events—the teacher's dis- 
pleasure, the criticism or ridicule of his classmates, 47 
ignominious showing in a competition, low marks, a tip 
to the office “to be talked to” by the principal, or a word 
to the parent who may still resort to the birch rod. In 
this welter of aversive consequences, getting the right 
answer is in itself an insignificant event, any effect of 
which is lost amid the anxieties, the boredom, and the 
aggressions which are the inevitable by-products of aver- 
sive control.2 g 

Secondly, we have to ask how the contingencies of 
reinforcement are arranged. When is a numerical opera- 
tion reinforced as “right”? Eventually, of course, the 
pupil may be able to check his own answers and achieve 
some sort of automatic reinforcement, but in the early 
stages the reinforcement of being right is usually accorde 
by the teacher. The contingencies she provides are far 
from optimal. It can easily be demonstrated that, unless 
explicit mediating behavior has been set up, the lapse © 
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only a few seconds between response and reinforcement 
destroys most of the effect. In a typical classroom, never- 
theless, long periods of time customarily elapse. The 
teacher may walk up and down the aisle, for example, 
while the class is working on a sheet of problems, pausing 
here and there to say right or wrong. Many seconds or 
minutes intervene between the child’s response and the 
teacher’s reinforcement. In many cases—for example, 
when papers are taken home to be corrected—as much as 
24 hours may intervene. It is surprising that this system 
has any effect whatsoever. ; 

A third notable shortcoming is the lack of a skillful 
program which moves forward through a series of progres- 
sive approximations to the final complex behavior de- 
sired. A long series of contingencies is necessary to bring 
the organism into the possession of mathematical be- 
havior most efficiently. But the teacher is seldom able 
to reinforce at each step in such a series because she 
cannot deal with the pupil’s responses one at a time. It 
is usually necessary to reinforce the behavior in blocks 
of responses—as in correcting a work sheet or page from 
a workbook. The responses within such a block must 
not be interrelated. The answer to one problem must 
not depend upon the answer to another. The number of 
stages through which one may progressively approach a 
complex pattern of behavior is therefore small, and the 
task so much the more difficult. Even the most modern 
workbook in beginning arithmetic is far from exem- 
plifying an efficient program for shaping up mathematical 
behavior. 

Perhaps the most serious criticism of the current class- 
room is the relative infrequency of reinforcement. Since 
the pupil is usually dependent upon the teacher for 
being right, and since many pupils are usually dependent 
upon the same teacher, the total number of contingencies 
which may be arranged during, say, the first four years, 
is of the order of only a few thousand. But a very rough 
estimate suggests that efficient mathematical behavior 
an this level requires something of the order of 25,000 
contingencies. We may suppose that even in the brighter 
Student a given contingency must be arranged several 
times to place the behavior well in hand. The responses 
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to be set up are not simply the various items in tables of 
addition, subtraction, multiplication, and division; we 
have also to consider the alternative forms in which 
each item may be stated. To the learning of such ma- 
terial we should add hundreds of responses concerned 
with factoring, identifying primes, memorizing series, 
using short-cut techniques of calculation, constructing 
and using geometric representations or number forms, 
and so on. Over and above all this, the whole mathe- 
matical repertoire must be brought under the control of 
concrete problems of considerable variety. Perhaps 50,000 
contingencies is a more conservative estimate. In this 
frame of reference the daily assignment in arithmetic 
seems pitifully meagre. 

The result of all this is, of course, well known. Even 
our best schools are under criticism for their inefficiency 
in the teaching of drill subjects such as arithmetic. The 
condition in the average school is a matter of wide- 
spread national concern. Modern children simply do not 
learn arithmetic quickly or well. Nor is the result simply 
incompetence. The very subjects in which modern tech- 
niques are weakest are those in which failure is most con- 
spicuous, and in the wake of an ever-growing incom- 
petence come the anxicties, uncertainties, and aggressions 
which in their turn present other problems to the school. 
Most pupils soon claim the asylum of not being “ready” 
for arithmetic at a given level or, eventually, of not hav- 
ing a mathematical mind. Such explanations are readily 
seized upon by defensive teachers and parents. Few 
pupils ever reach the stage at which automatic reinforce- 
ments follow as the natural consequences of mathemati- 
cal behavior. On the contrary, the figures and symbols 
of mathematics have become standard emotional stimuli. 
The glimpse of a column of figures, not to say an alge- 
braic symbol or an integral sign, is likely to set off—not 
mathematical behavior—but a reaction of anxiety, guilt, 
or fear, 

The teacher is usually no happier about this than the 
pupil. Denied the opportunity to control via the birch 
tod, quite at sea as to the mode of operation of the few 
techniques at her disposal, she spends as little time as 
possible on drill subjects and eagerly subscribes to phi- 
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losophies of education which emphasize material of 
greater inherent interest. A confession of weakness is her 
extraordinary concern lest the child be taught something 
unnecessary. The repertoire to be imparted is carefully 
teduced to an essential minimum. In the field of spelling, 
for example, a great deal of time and energy has gone 
into discovering just those words which the young child is 
going to use, as if it were a crime to waste one’s educa- 
tional power in teaching an unnecessary word. Even- 
tually, weakness of technique emerges in the disguise of 
a reformulation of the aims of education. Skills are 
minimized in favor of vague achievements—educating 
for democracy, educating the whole child, educating for 
life, and so on. And there the matter ends; for, unfor- 
tunately, these philosophies do not in turn suggest im- 
Provements in techniques. They offer little or no help 
in the design of better classroom practices. f 

_ There would be no point in urging these objections if 
improvement were impossible. But the advances which 
have recently been made in our control of the learning 
Process suggest a thorough revision of classroom practices 
and, fortunately, they tell us how the revision can be 
brought about. This is not, of course, the first time that 
the results of an experimental science have been brought 
to bear upon the practical problems of education. The 
modern classroom does not, however, offer much evi- 
dence that research in the field of learning has been re- 
spected or used. This condition is no doubt partly due 
to the limitations of earlier research. But it has been 
encouraged by a too hasty conclusion that the laboratory 
study of learning is inherently limited because it cannot 
take into account the realities of the classroom. In the 
light of our increasing knowledge of the learning process 
We should, instead, insist upon dealing with those reali- 
ties and forcing a substantial change in them. Education 
is perhaps the most important branch of scientific tech- 
nology. It deeply affects the lives of all of us. We can no 
longer allow the exigencies of a practical situation to sup- 
press the tremendous improvements which are within 


“teach. The practical situation must be changed. 


hich have to be answered 


. There are certain questions w! 
sm. What be- 


In turning to the study of any new organi 
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havior is to be set up? What reinforcers are at hand? 
What responses are available in embarking upon a pro- 
gram of progressive approximation which will lead to the 
final form of the behavior? How can reinforcements be 
most efficiently scheduled to maintain the behavior in 
strength? These questions are all relevant in considering 
the problem of the child in the lower grades. 

In the first place, what reinforcements are available? 
What does the school have in its possession which will 
reinforce a child? We may look first to the material to be 
learned, for it is possible that this will provide consider- 
able automatic reinforcement. Children play for hours 
with mechanical toys, paints, scissors and paper, noise- 
makers, puzzles—in short, with almost anything which 
feeds back significant changes in the environment and is 
reasonably free of aversive properties. The sheer control 
of nature is itself reinforcing. This effect is not evident 
in the modern school because it is masked by the emo- 
tional responses generated by aversive control. It is true 
that automatic reinforcement from the manipulation of 
the environment is probably only a mild reinforcer and 
may need to be carefully husbanded, but one of the most 
striking principles to emerge from recent research is that 
the net amount of reinforcement is of little significance. 
A very slight reinforcement may be tremendously effec- 
tive in controlling behavior if it is wisely used. 

If the natural reinforcement inherent in the subject 
matter is not enough, other reinforcers must be em- 
ployed. Even in school the child is occasionally permitted 
to do “what he wants to do,” and access to reinforce- 
ments of many sorts may be made contingent upon the 
more immediate consequences of the behavior to be 
established. Those who advocate competition as a use ul 
social motive may wish to use the reinforcements which 
follow from excelling others, although there is the difi- 
culty that in this case the reinforcement of one child 3s 
necessarily aversive to another. Next in order we mig 
place the good will and affection of the teacher, an 
only when that has failed need we turn to the use O! 
aversive stimulation. 

In the second place, how are these reinforcements tO 
be made contingent upon the desired behavior? There are 
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two considerations here—the gradual elaboration of ex- 
tremely complex patterns of behavior and the mainte- 
nance of the behavior in strength at each stage. The 
whole process of becoming competent in any field must 
be divided into a very large number of very small steps, 
and reinforcement must be contingent upon the accom- 
plishment of each step. This solution to the problem of 
creating a complex repertoire of behavior also solves the 
problem of maintaining the behavior in strength. We 
could, of course, resort to the techniques of scheduling 
already developed in the study of other organisms but in 
the present state of our knowledge of educational prac- 
tices, scheduling appears to be most effectively arranged 
through the design of the material to be learned. By 
making each successive step as small as possible, the fre- 
quency of reinforcement can be raised to a maxımum, 
while the possibly aversive consequences of being wrong 
are reduced to a minimum. Other ways of designing 
material would yield other programs of reinforcement. 
Any supplementary reinforcement would probably have 
to be scheduled in the more traditional way. 

These requirements are not excessive, but they are 
probably incompatible with the current realities of the 
classroom. In the experimental study of learning it has 
been found that the contingencies of reinforcement 
which are most efficient in controlling the organism 
cannot be arranged through the personal mediation of 
the experimenter. An organism is affected by subtle 
details of contingencies which are beyond the capacity of 
the human organism to arrange. Mechanical and electri- 
cal devices must be used. Mechanical help is also de- 
manded by the sheer number of contingencies whic 
may be used efficiently in a single experimental session. 
We have recorded many millions of responses from a 
single organism during thousands of experimental hours. 
Personal arrangement of the contingencies and personal 
observation of the results are quite unthinkable. Now, 
the human organism is, if anything, more sensitive to 
precise contingencies than the other organisms we have 
studied. We have every reason to expect, therefore, that 
the most effective control of human learning will require 
instrumental aid. The simple fact is that, as a mere Te- 
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inforcing mechanism, the teacher is out of date. This 
would be true even if a single teacher devoted all her 
time to a single child, but her inadequacy is multiplied 
many-fold when she must serve as a reinforcing device to 
many children at once. If the teacher is to take advantage 
of recent advances in the study of learning, she must have 
the help of mechanical devices. N 
The technical problem of providing the necessary 1n- 
strumental aid is not particularly difficult. There are 
many ways in which the necessary contingencies may be 
arranged, either mechanically or electrically. An inexpen- 
sive device which solves most of the principal problems 
has already been constructed. It is still in the experimen- 
tal stage, but a description will suggest the kind of in- 
strument which seems to be required. The device con- 
sists of a small box about the size of a small record plaver- 
On the top surface is a window through which a question 
or problem printed on a paper tape may be seen. The 
child answers the question by moving one or more sliders 
upon which the digits 0 through 9 are printed. The an- 
swer appears in square holes punched in the paper upon 
which the question is printed. When the answer has been 
set, the child turns a knob. The operation is as simple as 
adjusting a television set. If the answer is right, the kno 
turns freely and can be made to ring a bell or provide 
some other conditioned reinforcement. If the answer is 
wrong, the knob will not turn. A counter may be adde 
to tally wrong answers. The knob must then be reversed 
slightly and a second attempt at a right answer made. 
(Unlike the flash-card, the device reports a wrong an- 
swer without giving the right answer.) When the answet 
is right, a further turn of the knob engages a clutch 
which moves the next problem into place in the window. 
This movement cannot be completed, however, until the 
sliders have been returned to zero. 
_ The important features of the device are these: Re- 
inforcement for the right answer is immediate. The meré 
manipulation ef the device will probably be reinforcing 
enough to keep the average pupil at work for a suitab:e 
period each day, provided traces of earlier aversive CO” 
trol can be wiped out. A teacher may supervise an entire 
class at work on such devices at the same time, yet each 
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> child may progress at his own rate, completing as many 


problems as possible within the class period. If forced to 
be away from school, he may return to pick up where 
he left off. The gifted child will advance rapidly, but can 
be kept from getting too far ahead either by being ex- 
cused from arithmetic for a time or by being given special 
sets of problems which take him into some of the inter- 
esting bypaths of mathematics. 
_ The device makes it possible to present carefully de- 
signed material in-which one problem can depend upon 
the answer to the preceding and where, therefore, the 
most efficient progress to an eventually complex reper- 
toire can be made. Provision has been made for recording 
the commonest mistakes so that the tapes can be modi- 
fied as experience dictates. Additional steps can be in- 
serted where pupils tend to have trouble, and ultimately 
the material will reach a point at which the answers of 
the average child will almost always be right. $ 
If the material itself proves not to be sufficiently rein- 
forcing, other reinforcers in the possession of the teacher 
or school may be made contingent upon the operation 
of the device or upon progress through a series of prob- 
lems. Supplemental reinforcement would not sacrifice the 
advantages gained from immediate reinforcement and 
from the possibility of constructing an optimal series of 
steps which approach the complex repertoire of mathe- 
matical behavior most efficiently. 
A similar device in which the sliders carry the letters of 
the alphabet has been designed to teach spelling. In addi- 
tion to the advantages which can be gained from precise 
reinforcement and careful programming, the device will 
teach reading at the same time. It can also be used to 
establish the large and important repertoire of verbal 
relationships encountered in logic and science. In short, 
it can teach verbal thinking. As to content instruction, 
the device can be operated as a multiple-choice self-rater. 
Some objections to the use of such devices in the 
classroom can easily be foreseen. 
that the child is being treated as a mere animal and that 
an essentially human intellectual achievement is being 
analyzed in unduly mechanistic terms. Mathematical 
behavior is usually regarded, not as a repertoire of re- 
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as evidences of mathematical ability or the exercise of the 
power of reason. It is true that the techniques which 
are emerging from the experimental study of learning are 
not designed to “develop the mind” or to further some 
vague “understanding” of mathematical relationships. 
They ate designed, on the contrary, to establish the very . 
behaviors which are taken to be the evidences of z 
‘ 


| 
sponses involving numbers and numerical operations, ] 


mental states or processes. This is only a special case of 
the general change which is under way in the interpreta- 
tion of human affairs. An advancing science continues to 
offer more and more convincing alternatives to traditional 
formulations. The behavior in terms of which human 
thinking must eventually be defined is worth treating 1M 
its own right as the substantial goal of education. 

Of course the teacher has a more important function — 
than to say right or wrong. The changes proposed would 
free her for the effective exercise of that function. Mark- 
ing a set of papers in arithmetic—“Yes, nine and six are 
fifteen; no, nine and seven are not eighteen”—is beneath | 
the dignity of any intelligent individual. There is more ` 
important work to be done—in which the teacher’s rela- 
tions to the pupil cannot be duplicated by a mechanical 
device. Instrumental help would merely improve these 
relations. One might say that the main trouble with 
education in the lower grades today is that the child is 
obviously not competent and knows it and that the — 
teacher is unable to do anything about it and knows that 
too. If the advances which have recently been made in 
our control of behavior can give the child a genuine com- 
petence in reading, writing, spelling, and arithmetic, 
then the teacher may begin to function, not in lieu of 4 
cheap machine, but through intellectual, cultural, and 
emotional contacts of that distinctive sort which testify 
to her status as a human being. 

_ Another possible objection is that mechanized instruc- 
tion will mean technological unemployment. We need 
not worry about this until there are enough teachers tO 
80 around and until the hours and energy demanded of | 
the teacher are comparable to those in other fields of ` 
employment. Mechanical devices will eliminate the more 

tiresome labors of the teacher but they will not neces- 
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sarily shorten the time during which she remains in 
contact with the pupil. 

A more practical objection: Can we afford to mecha- 
nize our schools? The answer is clearly yes. The device I 
have just described could be produced as cheaply as a 
small radio or phonograph. There would need to be far 
fewer devices than pupils, for they could be used in rota- 
tion. But even if we suppose that the instrument eventu- 
ally found to be most effective would cost several hundred 
dollars and that large numbers of them would be re- 
quired, our economy should be able to stand the strain. 
Once we have accepted the possibility and the necessity 
of mechanical help in the classroom, the economic prob- | 
lem can easily be surmounted. There is no reason why 
the school room should be any less mechanized than, 
for example, the kitchen. A country which annually pro- 
duces millions of refrigerators, dish-washers, automatic 
washing-machines, automatic clothes-driers, and automa- 
tic garbage disposers can certainly afford the equipment 
necessary to educate its citizens to high standards of com- 
petence in the most effective way. 

There is a simple job to be done. The task can be 
stated in concrete terms. The necessary techniques are 
known. The equipment needed can easily be provided. 
Nothing stands in the way but cultural inertia. But what 
is more characteristic of America than an unwillingness 
to accept the traditional as inevitable? We are on the 


‘threshold of an exciting and revolutionary period, in 


which the scientific study of man will be put to work in 
man’s best interests, Education must play its part. It 
Must accept the fact that a sweeping revision of educa- 
tional practices is possible and inevitable. When it has 
done this, we may look forward with confidence to a 
school system which is aware of the nature of its tasks, 
secure in its methods, and generously supported by the 
informed and effective citizens whom education itself 
will create. 
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Teaching Machines: An Application 
of Principles From the Laboratory* 


Haryard University 


Much has been said of teaching machines recent y A 
but the emphasis has tended to be on the gadgets ra i, 
than on the much more significant development of a n ka 
technology of education initiated by B. F pees 
(1954). The technology does use a device called a ten A 
ing machine which presents a finely graded yaa 
problems and provides immediate “reward” or rein o on 
ment for the student’s correct answers. But emphasis it 
machines has tended to obscure the more it ta 
facets of the new technology based on application Pe 
principles from the laboratory. The machines of weal 
are not necessarily better than those of yesterday. Inde of 
adequate machines could have been built hundrede og 
years ago. The movement today is not simply the mie E 
nization of teaching, but instead the development 0! Z 
new technology—a_ behavioral engineering of teachin 
procedures. 


The history of unsuccessful teaching machines illus 


James G. Hortan 
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trates the relatively greater importance of the technique 
as opposed to the gadgets. The first teaching machine 
was patented 93 years ago. There have since been a series 
of patents and a promising burst of activity by Sidney 
Pressey and his associates (1926). None of these early 
efforts really caught hold. But during this period in which 
the idea of mechanized teaching has been latent the 
science of behavior has developed principles which per- 
mit extremely precise control of behavior. This new 
technology is not only the so-called automation of teach- 
ing but is an attempt to obtain the kind of behavioral 
control shown possible in the laboratory. : 
We have, of course, seen other practical applications 
of scientific psychology. We are all familiar with the 
development of a technology of testing which permits 
placing an individual in situations suited to his abilities. 
We are also familiar with another technology called hu- 
man engineering, which fits machines and jobs to the 
capacities of man. One places a man in a job that suits 
him; the other alters the job to suit the man; neither 
attempts to alter or control man’s behavior. 
For years in the laboratory we have controlled the 
behavior of experimental subjects—both animal and hu- 
man—by a widening array of principles and techniques. 
The new technology of education is the application © 
behavioral laws in modifying oF controlling behavior. 
Such a technology became possible with the realization 
that we are actually referring to a verbal repertoire 
(Skinner, 1957) controlled by the same laws as other 
behavior. The old, defunct explanatory concepts of 
knowledge, meaning, mind or symbolic processes have 
never offered the possibility of manipulation or control; 
Pat behavior, verbal or otherwise, can be controlled with 
‘ase and precision. X 
WEE machines are not the essential or defining as- 
pects of this technology, they do play an important role 
in providing some of this fine control the technology 1e- 
quires. We will now examine several machines and notice 
the advantages they offer. : 
At Harvard there is a self-instruction room with ten 
ooths, each containing a machine. The student gets one 
Set of material from the attendant and places it in the 
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machines. He closes the machine and begins his staat 
This machine presents one item of material at a a 
The subject reads the statement which has one or maA 
words missing and he completes the statement by ecu 
in the answer space. He then raises the lever and a mal 
shutter opens revealing the correct answer, and simu E 
ously his answer is moved under glass where it can aa 
read and compared with the new-exposed correct ans thel 
After comparing his answer with the correct answer b 
student indicates to the machine, with an appropri ct 
movement of the lever, whether his answer was pee 
or incorrect and the next item appears in the wine 
All items answered wrong are repeated after he ap ea 
the set of items. Correctly answered items are no 
eated. > Jes 
: A critical feature of the machine is that it provid 
immediate reinforcement for correct answers. Being oA 
tect is known to be a reinforcer for humans E 
1943). In machine teaching reinforcement is immedia 4 
We know from laboratory work that a delay ey 
response and its reinforcement of a few seconds ‘id 
greatly reduce the effectiveness of the reinforceme a 
Adult humans have developed behavior which Sever 
mediate small delays; nevertheless, any delay makes 1 
forcement less effective. ork- 
Although other techniques such as programmed sting” 
books (Homme and Glasser, 1958), “self-correcti s 
homework (Diederich, 1960), and flash cards are son el 
times used in this new behavioral technology, they rr, 
less control. Teaching machines eliminate ope in 
forms of responses which would also be successful i g 
obtaining the right answer. For example, the pene 
machine insures that the student answer before peek? 
at the indicated answer. There is a strong egg ieee a 
glance ahead with only a poorly formulated, unwri 
answer when flashcards are used. m 
This write-in machine is a prototype of the most CO he 
mon machiné. There is another machine used for teac a 
ing young children material which consistently has 
single possible answer, In the machine the constructe! 
answer is automatically compared with the true anse 
The child is presented a problem, perhaps a statemen 


) density of correct items is necessary 
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such as 2+2 = , and he must provide the “4.” 
By moving a slider appropriately he can insert the 4 into 
the answer space. He then turns the crank, and the next 
item appears immediately, and therefore, immediate re- 
inforcement is provided. 

Both of the machines described thus far require the 
student to compose the answer. A machine for a less 
mature organism who cannot yet compose an answer can 
be used for teaching preschool children (Hively, 1960). 
In this machine there is a large top window and three 
small windows. In the large window there is some sort 
of problem, and in the three smaller windows, there are 
three alternative choices. When the correct choice is 
made, the next frame is presented. 

Enough of machines. They should not be allowed to 
obscure the truly important feature of the new tech- 
nology, namely, the application of methods for behav- 
ioral control in developing programs for teaching. We 
need to say no more about the well-known principle of 
immediate reinforcement. Our second principle is also 
well known. Behavior is learned only when it is emitted 
and reinforced. But in the classroom, the student per 
forms very little, verbally. However, while working with 
a machine, the student necessarily emits appropriate be- 
havior and this behavior is usually reinforced since the 
material is designed so that the student is usually correct. 
Not only is reinforcement needed for learning, a high 
because material 


Which generates errors is punishing. Laboratory exper 


ments (Azrin, 1956) have shown that punishment lowers 
the rate of the punished behavior. In our experience with 
teaching machines we have also observed that students 
stop work when the material is so difficult that they make 
Many errors, Furthermore, they become irritated, almost 
aggressive, when errors are made. 

The third important principle is that of gradual pro- 
Btession to establish complex repertoires. A visitor once 
asked if Skinner had realized that pigeons were so smart 
ea he began using them as subjects. The answer given 

Y a helpful graduate student was that they weren't so 
Smart before Skinner began using them. And indeed they 


Weren’t. The behavior developed in many experiments 
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is like that developed in the classroom. Both are complex 
operants. Both require a careful program of gradual p10- 
gression. We cannot wait for a student to describe the 
content of a psychology course and then reinforce the 
performance; nor can we wait for a pigeon to emit such 
an improbable bit of behavior as turning a circle, facing A 
a disk on the wall, pecking it if lit, and then bending 
down to a new exposed food tray and eating. F 

When developing a complex performance in a pigeon, 
we may first reinforce simply the behavior of approaching 
the food tray when it is presented with a loud click. Late? 
the pigeon learns to peck a key which produces the click , 
and the food tray. Still later, he may learn to peck this 
key only when it is lit, the peck being followed by the 
loud click and approach to the food tray. In the next 
step, he may learn to raise his head or hop from one foot 
to another, or walk a figure eight, in order to produce the 
lighted key which he then pecks; the click follows; and 
he approaches the food tray. 

This principle of gradual progression runs through 
many of the teaching machine techniques. Both human 
and avian scholars deserve the same careful tutorage. The 
teaching machine program moves in very finely graded 
steps, working from simple to an ever higher level of com 
plexity. Such a gradual development is illustrated 1? 
Table 1 by a few items taken from a psychology program \ 
(Holland ‘and Skinner). Í ; 

The principle of gradual progression serves not simply 
to make the student correct as often as possible, but it 
is also the fastest way to develop a complex repertoire 
In fact, a new complex operant may never appeat except 
through separately reinforcing members gf a graded series | 
(Keller and Schoenfield, 1950). Only this way can wel 
quickly create a new pattern of behavior. The pigeo® 
would not have learned the compléx sequence necessi, 
to receive the food if he had not learned each step in a 
proper order. Obviously a child can’t begin with advance 
mathematics, but neither can he begin with 2 + =e 


even this is too complex and requires a gradual progres” 
sion. 


Our fourth principle is, in a sense, another form q 
gradual progression—one which involves the gradua 
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TABLE 1 


Items from the psychology program ( 
These items illustrate the gradual development of a 


Item 


. Performing animals are some- 


times trained with “rewards.” 
The behavior of a hungry ani- 
mal can be “rewarded” with 


. A technical term for “reward” 


is reinforcement. To “reward” 
an organism with food is to 
on . it with food. 


. Technically speaking, a thirsty 


organism can be .....--+++* 
with water. 


A school teacher is likely, 
whenever possible, to dismiss 
a class when her students are 
rowdy because she has been 
Hedy .... by elimination 
of the stimuli arising from a 
rowdy class. 


The teacher who dismisses a 
class when it is rowdy causes 
the frequency of future rowdy 


behavior to (1) .....0-6+2+ 
Since dismissal from class is 
Probably a(n) (2) .....++++9* 


for rowdy children. 


If an airplane spotter never 
Sees the kind of plane he is to 
Spot, his frequency of scanning 
the sky CD) rn T 
other worde, his “looking” 
behavior is (2) 0.2... 0e 


Correct Answer 
Food 


Reinforce 


Reinforced 


Reinforced 


(1) Increase 
(2) Reinforcement 


(1) Decreases 
(2) Extinguished 
(or: Not 

Reinforced) 
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‘Holland and Skinner). 
new concept. 


Percentage 
of Students 
Giving 
the Answer 
96 


100 


100 


92 


86 


94 
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This method will be illustrated with some neuroanatomy 
material.1 A fully labelled cross section of the medulla 
oblongata is placed before the student while he works 
with a large set of items pertaining to the spatial arrange 
ment of the various structures. For example, “posterior 
to the cuneate nuclei are the —__—.”” The answer 18: 
“the cuneate fasciculi.” 

After many such items, he begins another set and has 
another picture before him, but now the structures are 
only labelled with initials. A new set of items again asks 
a long series of questions pertaining to the spatial pos! 
tion of the various structures. For example, “between the | 
gracile and the trigeminal nuclei are ——.” The am 
swer is the “cuneate nuclei.” 

After many more items, he proceeds to a new set and 
the next picture. This time the picture is unlabelled- 
Again, he goes through a series of new items, not simple 
repetition of the previous ones, but pertaining to the 
same program of the spatial location of the different 
structures. This set is followed by still another, but with 
no picture at all. He is now able to discuss the spatial 
position of the structures without any visual representa- 
tions of the structures before him. In a sense, he has his 
own private map of the medulla. He may further demon- 
strate his newly acquired ability by accurately drawing the 
medulla. 

The neuroanatomy example is an elaborate example of 
fading. Fading is also applied in a more simple form 1 
constructing verbal programs without pictorial displays. 
single item may in one sentence give a definition or 4 
general law and a second sentence in that same item a? 
example in which a key word is omitted. This would be 
followed by a new example in the next frame, but with 
the definition or law lacking. 

This brings us to our fifth principle, control of the 
student’s observing and echoic behavior. In the classroom 
the student is often treated as though he were some kin 


withdrawal of stimulus support. This we shall call fading: | 


* This material was prepared by D. M. Brethower in CO} 


i 
laboration with the present author and is being used at Harvard 
for research purposes. 
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pod la sect of information who can sop up infor- 
pcre ae? by the teacher, written on the blackboard, 
ere y films. But all of these are effective only 
ae sed gee has some behavior with respect to 
oo Je must listen carefully, or read carefully, 
ie gaging in usually covert echoic behavior. 
‘ee — of classroom techniques is often cred- 
= foe ee or poor “concentration.” It has 
ee G eid, 1953) that if a discrimination is to 
aE te observing behavior must first be 
tise ores € have further found (Holland, 1958) 
ME i kine behavior or, speaking loosely, “attention” 
This bee si e same forms of control as other behavior. 
ce oe of observing behavior is of prime impor- 
the te an the student becomes very “inattentive” in 
mp om em the teaching material flows on; but with 
lovee ang he moves ahead only as he finishes, an item. 
en ike Fa participation result in nothing more 
Unde chine sitting idle until the student con- 
£ aiei, honest a more subtle aspect to the control 
he ng ehavior than this obvious mechanical one. 
Senin oo the examples we have seen, success in an- 
ful ae e problem only depends on the student’s care- 
servation of the material in front of him at the 


mo : > : 
ment. This may be illustrated by more material from 
showing stimulus gen- 


tudent while he works 
may complete a state- 


the psychology 
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ealon data is in front a the s 
meee In the program he 
either ax the following: “As the wave length changes in 
E a from the wave length present during re- 
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behav s “decreases.” The item serves 0 

| ior of observing the data. Of course, many m 


nly to control the 
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h items are used to discuss the same data. 
tion extends to the 


events may have a common effect. 
will vary wi priate to different deprivations 

ty with ————. deprivations.” The answer is “two” 
hoice of the omission 
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serves to insure a careful reading of the item. Only those 
parts of an item which must be read to correctly com- 
plete a blank can safely be assumed to be learned. —, 

Our sixth principle deals with discrimination training. 
In learning spoken languages, for example, it is necessary 
to be able to identify the speech sounds. A student may 
listen to a pair of words on a special phonograph which 
repeats the passage as many times as he desires. The 
visual write-in machine instructs him to listen to a spe 
cific passage. For example, the student may hear two 
words such as: “sit, set.” He listens as many times as he 
needs, then writes the phonetic symbols in the write-in 
machine and operates the machine, thereby exposing the 
true answer and providing immediate reinforcement fot 
his correct discrimination. : 

However, little academic education is simple discrimi- 
nation. More often, it is abstraction or concept forma- 
tion, An abstraction is a response to a single isolated 
property of a stimulus. Such a property cannot exist 
alone. Redness is an abstraction. Anything that is red 
has other properties as well—size, shape, position in 
space, to name a few. There are red balls, red cars, red 
walls, The term “red” applies to all of them but not to 
green balls, blue cars, or yellow walls. To establish an 
abstraction (Hovland, 1953), we must provide many €x- 
amples; each must have the common property, but 
among the various examples there must be a wide range 
of other properties. This is best illustrated by examples 
from a preverbal machine. 

These examples are from a program? which teaches 4 
child to respond to the abstract property of form. In 
each item, the upper figure is the sample and the lower 
three are the alternatives. While developing a program 
for establishing an abstraction we remember our earlier 
principles, and move through a gradual progression. For 
the first several items there is a sample and a single 
match, the other two being blank. The sample and its 
match are exacily alike at this stage. 

_ After many such items, we would begin to have others 
in which the sample and its match again correspond i” 
size, color and form, but an additional incorrect alterna 


? The program was prepared by B. F. Skinner, 
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tive has been added which differs from the sample in 
all these aspects. Later we move on to frames with three 
choices, again the sample and its match corresponding 
exactly, Next, the sample and the match may differ in 
some property such as color or size. It is essential that 
the program contain many items among which the sam- 
ple and correct match differ in all properties except the 
one providing the basis for the abstraction. Otherwise the 
abstraction will be incomplete because the extraneous 
property will share some of the control over the abstract 
Tesponse, 

As we move on with additional examples, the sample 
and the correct match differ both in color and in size, an 
the incorrect alternatives are beginning to share some of 
the extraneous properties with the sample. The student 
continues with many such problems in which the only 
common property between the sample and the correct 
match is the shape, regardless of size and color. ke 
now our abstraction may be incomplete. We have kep 


the figures in only one orientation. Therefore, we also 


have a series in which the samples are rotated. i 
A great deal of academic education consists of one 
to teach abstractions. Concepts such as force, tenca 
ment, supply-and-demand, freedom, and many, re 
Other possible examples are all abstractions. sig pe Se 
in the academic setting, the student seldom adequatey 
forms abstractions. The trigonometry student pong 
uses triangles with the right angle as one of shes ‘igh t 
angles, If the triangle is rotated 90°, so that the oe it 
angle is upward, the student often does not aor d 
as a right triangle. Neither is an abstraction ris it 
simply by learning a definition. The psychology stu en 
Who learns the definition of reinforcement 1m oot 
terms and is acquainted with a laboratory eamp H 
food reinforcement, may not realize the horrible cons z 
quences of sending his girl friend flowers to end an argu 
Ment. Thus, in the psychology program, n w T a 
new concept, we follow the pattern in the preverbal ex- 


5 hich 
ample. A wide range of examples are analyzed whic 
iffer in as many aei as possible, each still having the 
common property which characterizes tie it of 
The last principle I shall discuss 15 really a question 
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methodology which has served so well in the laboratory. 
This principle is to let the student write the program. A 
few years ago, a cartoon was published in the Columbia 
Jester. The rat leaning on a bar is saying to the other rat: 
“Boy, do we have this guy conditioned. Every time I 
press the bar down, he drops a pellet in.” 

Although said in jest, it is true that the rat controls 
the experimenter’s behavior. When interesting things are 
observed about the rat’s behavior, the control circuits are 
rewired to investigate the interesting new facet of behav- 
ior. In a sense, the rat is wiring the control circuit. Simi- 
larly, the behavioral engineer who prepares good teaching 
machine material must be under the control of the stu- 
dent's responses. When the student has trouble with part 
of a program the programmer must correct this. The stu- 
dent’s answers reveal ambiguities in items; they reveal 
gaps in the program and erroneous assumptions as to the 
student’s background. The answer will show when the 
program is progressing too rapidly, when additional 
prompts are necessary, or when the programmer should 
try new techniques. When unexpected errors are made, 
they indicate deficiencies not in the student but in the 
program. 

The most extensive experience with this principle of 
modifying the program to fit the student has been at 
Harvard (Holland, 1959) with the psychology program. 
In 1958, we had a program consisting of forty-eight disks 
or lessons of twenty-nine frames each. After using the 
program and making a detailed, item-by-item analysis of 
the students’ answers we diagnosed the particular defici- 
encies in the program and revised it accordingly. The 
program was also extended to cover a larger amount of 
subject matter and, in 1959, it consisted of sixty disks 
You have already seen a few items from the course After 
using the revised material in 1959, we evaluated the ex- 
tent of its improvement. Table 2 shows the percentage 
aaa on the first twenty disks for each of the two 

The revision eliminated about half the errors. The last 
column of the table gives percentage of improper self- 
scoring by the students. Revision also cut these scoring 
errors approximately in half. Furthermore, the revision 
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TABLE 2 


A comparison of the student’s errors in 
using the revised (1959) and unrevised 
(1958) program in psychology. 


Per cent Items 
Per cent Improperly Scored 
Errors by Students 
1958 20.1 3.6 
1959 11.0 14 


decreased the time required to complete the material. 
Although the second year’s material had more disks— 
sixty as opposed to forty-eight—it actually required the 
average student about one hour less to complete the 
work than it had required for the shorter first version. 
Frequency distributions on the median times in minutes 
for completion of the various disks are shown in Fig. 1. 


35 — |959 
=-=- 1958 


25 


PERCENTAGES OF DISKS 
a 


65 105 145 185 225 265 305 
MEDIAN TIMES OF ALL CYCLES (MINUTES) 


Fig. 1. Here are the frequency distributions for the median 
times to complete disks or “lessons” for the revised and unre- 
vised psychology program. Raw frequencies were converted to 
percentages to equate the area under the curves. 

These are the times required for the median student to 
move through each set of material answering every item 
once and to repeat the items answered incorrectly. Notice 
the considerable time required for many disks in the first 
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year’s material. Primarily this was because students re- < >=% 


peated the larger number of items missed in the first 
cycle. 2 

But the improved material provided faster perform- 
ance, even when the delay due to repetition of incorrectly 
answered items is not considered. The frequency dis- 
tributions for the first cycle only are provided in Fig. 2. 


PERCENTAGES OF DISKS 


6.5 10.5 14.5 18.5 22.5 
MEDIAN TIMES OF FIRST CYCLES (MINUTES) 


Fig. 2. Here are the frequency distributions for the median 
times to complete only the first cycles for the revised and unre- 
vised psychology program. Raw frequencies were converted to 
percentages to equate area under the curves. 


These data exclude the time used in repeating items. 
Here too the revision produced slightly more rapid prog- 
Tess. 

Such careful tailoring of material to fit the student is 
impossible with most teaching techniques. With teaching 
machines, as in no other teaching technique, the pro- 
grammer is able to revise his material in view of the 
students’ particular difficulties. The student can write 
the program; he cannot write the textbook. 

We have seen that the rinciples evolved from the 
laboratory study of behavior lave provided the possibility 
for the behavioral engineering of teaching. This new 
technology is taoroughly grounded in some of the better 
established facts of behavioral control. The future of 
education is bright if persons who prepare teaching ma- 
chine programs appreciate this, and appropriately educate 
themselves in a special, but truly not esoteric, discipline. 
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But it is vital that we continue to apply these techniques 
in preparing programs. The ill-advised efforts of some of 
our friends, who automatize their courses without adopt- 
ing the new technology, have an extremely good chance 
of burying the whole movement in an avalanche of 
teaching-machine tapes. 
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Skinner’s Theory of 
Teaching Machines* 


Davin ZEAMAN 
University of Connecticut 


One of the differences between the natural and the 
social sciences is that in the natural sciences, each suc- 
ceeding generation stands on the shoulders of those that 
have gone before, while in the social sciences, each gen- 
eration steps in the faces of its predecessors. 

Whether psychology is a natural science or an un- 
natural one may be a question, but given a practical psy- 
chological problem of training devices to solve, there is 
no question at all that we have a choice of two strategies 
here, the shoulder or the face. 

If it is the shoulder, where should the constructor and 
the programmer of human training devices look in the 
scientific literature of psychology for experimental data 
relevant to his task? 

Any answer to this question implies a theory of train- 
ing devices. It is in this loose but yet important sense 
that the word theory is here used. 

Skinner has a theory, if I read him rightly, and it says 
the appropriate reference class of experiments, the ap- 
propriate experimental paradigm, for construction and 


* This article originally appeared in Eugene Galanter (Ed.), 
Automatic Teaching: The State of the Art. New York: John 
Wiley & Sons, Inc., 1959, Chapter 14. It is used here with the 
permission of the author and the publisher. 
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programming teaching machines is that of free operant + 


conditioning. Others, in particular those using the mem- 
ory drum, are held to be inappropriate. 

Skinner’s resolution of the shoulder-face question is, 
happily, the shoulder, but the shoulders are too ivy- 
league-ish—too narrow. 

It is the counter-thesis of this talk, that existing teach- 
ing machines (and those likely soon to be built) Tepre- 
sent complex admixtures of at least three basic experi- 
mental paradigms, free operant, controlled operant, and 
classical conditioning, and that of these, the free operant 
may be the least pertinent. Furthermore, of all the com- 
monly used apparatuses for experimental study of learn- 
ing that one which most closely resembles one of Skin- 
ner’s teaching machines (the Write-in Model) is the 
memory drum. 

Since there is something less than universal agreement 
on classification of learning paradigms, let me ask for a 
temporary convention in the use of the words “controlled 
operant, free operant, and classical conditioning.” 

All three refer to experimental arrangements of ob- 
servable variables that a machine can handle. The com- 
mon elements of all three include two stimulus events 
S; and So, occurring in that order, and at least two re- 
sponses arbitrarily designated as correct and incorrect. 

By “controlled operant” or “discrete trial operant” is 
meant those experiments incorporating the following ac- 


quisition features: (keeping the Graham-Gagne Runway + 


experiments in mind will help you here). 

The acquisition trial begins with the presentation of 
a stimulus situation S, which lasts until the subject makes 
a correct response—R. One and only one correct response 
R occurs per trial because the correct response terminates 
S,. It also brings Sa, which in this case is a reinforcing 
stimulus. S does not elicit R. The timing arrangements 
are partially subject controlled, partially experimenter. 
The time interval between S, and S, is subject controlled 
since Sp is contingent upon R, but the intertrial interval, 
that is the Sọ — S, interval is usually under the experi- 
menter’s control. 

Acquisition is measured by S-R latency or the condi- 
tional probability of R given S,. 


bet 
; 
l 
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Illustrative apparatuses for controlled operant condi- 
tioning include the runway, Miller-Mowrer Shuttle Box, 
Retractible Bar Skinner Box, the Single-door Lashley 
Jumping Stand, and also if correction technique is used, 
the Simple T-Maze, Discrimination Boxes and the Two- 
door Jumping Stand. 

In the free operant or ordinary Skinner box situation, 
similar features are present with some exceptions. The 
initial stimulus situation S, is of prolonged duration, al- 
lowing many correct responses to occur, each one rein- 
forced during continuous acquisition by the contingent 
reinforcing stimulus Sə. This of course permits a rate 
measure to be taken. 

Both the free and controlled operant paradigms, then, 
display an SRS sequence, with the second S contingent 
upon R. The big difference lies in the number of correct 
responses allowed per trial, only one in the controlled 
operant (since the correct response eliminated S, as well 
as bringing S2), but many correct responses in the free 
operant. 

What about timing differences? Is one self-paced and 
the other not? If one accepts Estes’ analysis of the free 
operant, the animal turns S, on and off by himself, even 
though the experimenter leaves it on. Each reinforcing 
stimulus Sə turns over the S, and the subject gets a fresh 
look at S,. In this analysis, the free operant is self-paced, 
in the sense that the subject controls the intertrial inter- 
val, the time between S, and S,. 

This is to be contrasted with the controlled operant 
in which the subject waits for the experimenter to turn 
on §,. It is not completely self-paced. 

It is important to note here that if you accept this 
analysis and regard the free operant as self-paced, it is 
only so when the S,’s remain the same. 

If the experimenter has a number of different S,’s to 
be connected to a number of different correct responses, 
free operant techniques can be used, but then it is no 
longer completely self-paced. The experimenter changes 
the problem. The inter-problem interval is not subject 
controlled, and it is this interval that we are usually in- 
terested in with teaching machines. 

Now finally to the classical conditioning paradigm. 
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The previous two paradigms were SRS sequences with a 
contingency. This one is an SSR sequence with no con- 
tingency. To fulfill the paradigm, the first stimulus S, 
must not elicit R as strongly as you would wish, whereas 
the second stimulus Sə must elicit R, the correct response. 

This is a highly operational, non-theoretical specifica- 
tion of the classical conditioning paradigm. It is not as- 
serted that S, be an innate elicitor of R, nor even a strong 
stimulus. Whether such a paradigm always provides the 
necessary or sufficient conditions for learning is an empiri- 
cal question not easily resolved, given the existing litera- 
ture. It is, however, a paradigm that can be used in con- 
struction of training devices, and it looks operationally at 
least to be different from free and controlled operant con- 
ditioning, in not having a contingent Sp and in requiring 
S2 to elicit the correct response. 

In the matter of timing, classical conditioning resem- 
bles the controlled operant in that the experimenter con- 
trols the inter-trial interval (the Sp, S, gap) but classical 
conditioning differs from both free and controlled oper- 
ants in also giving control over the S, — Sx interval to the 
experimenter. 

With these three paradigms specified in this way, let 
us take a look at three pieces of apparatus, a memory 
drum set up for paired associates, Skinner’s Write-in 
Model teaching machine and Skinner’s Arithmetic ma- 
chine. 


First the paired associates experiment. On the first » 


trial around, there is ordinarily no anticipation. The sub- 
ject sees the S, syllable and reads the correct associate. 
This fits the classical conditioning paradigm. It is an SSR 
sequence, no contingency, and all stimulus intervals 
under control of the experimenter. 

On the second and subsequent trials, the possibility 
of operant conditioning enters. If the subject correctly 
anticipates the associate, he is reinforced by the match 
of his anticipated answer and the forced answer. Since 
this matching’ or knowledge of correct Tesults, is con- 
tingent upon a correct anticipation, elements of a con- 
trolled operant are present. j 

If the subject gives the wrong response, the contingent 
reinforcement is not present, and the trial tesembles an 


» 
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extinction trial of a controlled operant. The trial ends, of 
course, with a classical conditioning feature, a forcing of 
the correct response, elicited by the appearance of the 
associate. 

If no overt response is given at all, the trial becomes a 
simple classical conditioning trial again. 

Thus there are three kinds of trials possible with a 
paired associates machine, a simple classical trial, a con- 
trolled operant acquisition trial with a terminal classical 
feature, or a controlled operant extinction trial with a 
terminal classical feature. The timing of these trials is 
almost exclusively that of a classical conditioning, how- 
ever both the S, — Sa interval and the S — S; interval are 
experimenter controlled. To my knowledge there are no 
animal experiments with trials at all like these, although 
I believe they might be arranged. 

Consider now Skinner's Write-in Model learning ma- 
chine. It presents a written question or an incompleted 
sentence or paragraph in an aperture, and then waits for 
the subject to write in his answer or completion. Follow- 
ing this the subject operates the machine to present the 
correct written answer to be compared with his own. The 
next problem appears when the subject indicates to the 
machine that he is ready. 

We analyze the machine in the terms used before. As 
with the memory drum there are three kinds of trials, 
depending upon whether the subject makes no response, 
makes the correct response, or makes an incorrect re- 
sponse. 

If no response is made, the trial looks like a classical 
conditioning paradigm or SSR. The question stimulus 
S, is presented. Then the answer stimulus follows, and 
elicits the answer response. No contingency, just a forced 
response like classical conditioning. 

If the correct response is made, an instrumental or 
operant feature emerges. The correct answer given in 
anticipation of Sə is reinforced by a matching of the 
correct results. This reinforcement is of course contin- 
gent, and makes the trial resemble the free or controlled 
Dperant. But since there is only one correct response 
made, this aspect of the trial looks most like the con- 


trolled operant. 
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If an incorrect response is made, the contingent match- 
ing reinforcement is absent and we have elements of a 
controlled operant extinction trial. The trial is terminated 
however with the forced response characteristic of the 
classical procedure. 

The high similarity of these three kinds of trials with 
those of the paired associates memory drum experiment 
is obvious. Yet there is some difference. A difference in 
timing. With the usual paired associates procedure, all 
stimulus intervals are experimenter controlled. With ‘the 
Write-in Model, in contrast, the S, — S, interval is sub- 
ject controlled (that is, the subject can give himself as 
much time as he wants before presenting S) and the 
interproblem interval is also subject controlled. It is in- 
deed a self-paced device. 

The subject’s control of the timing of the Sẹ stimulus 
is by no means new: it is a property of free and controlled 
operant procedures in general, but the subject’s control 
of the interproblem interval is relatively novel. It is not 
found in either the usual controlled operant experiment, 
or in the usual free operant, nor of course, in classical 
conditioning. Some recent experiments on the role of 
observing responses in animal discrimination have incor- 
porated an analogous type of paradigm. Wyckoff had 
pigeons turning on their own discriminative stimuli by 
stepping on a treadle. His work should be relevant to 
this machine. And other work may be relevant too. 

Since there have been so many memory drum experi- 
ments done, and since it is no engineering trick at all 
to convert the usual memory drum to an entirely self- 
paced apparatus, like Skinner’s Write-in Model, I would 
be surprised if a literature survey failed to turn up any 
self-paced paired associates data. Of course when a sub- 
ject rifles through a deck of flash cards at his own time, 
the paradigm for this primitive machine is the same as 
that of Skinner’s Write-in Model. 

The last device to be inspected is a modification of one 
of Skinner’s Gevices which we have also constructed and 
called the “Arithmachine.” It presents visual material 
and allows the subject to make any of a large number of» 
numerical responses, one of which is correct. The re- 
sponse is the movement of sliders and the turning of a 


= 
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` crank. If the sliders have been positioned correctly, each 


slider position representing a number, the turning of the 
crank will bring reinforcement and the next problem. 

The analysis. This machine operation looks much more 
like the controlled operant procedure than does the 
Write-in Model. The S, or question is presented, a re- 
sponse is waited for, and a contingency set up. If the 
response is correct, the subject is rewarded by the com- 
plete turning of the crank and whatever other reinforcing 
signals the experimenter wants. There are no forced re- 
sponses of a classical nature at the end of the trial. The 
correct response not only brings reinforcement, it elimi- 
nates S,, the question stimulus, thus allowing only one 
correct response per trial, the hallmark of the controlled 
operant. 

Timing is not completely self-paced. The S, — S, inter- 
val is subject controlled, but the interproblem interval is 
not. The problems come in massed sequence. As soon 
as the subject finds out that he has been right on the last 
problem, the new one stares up at him. 

Strength of acquisition is inferred with this apparatus 
from latency or probability of response, as it is for the 
other machines mentioned. Free rate of operant response, 
in the presence of a prolonged S, is not taken here, nor is 
it a measure taken for any teaching machine that I know 
of. 

This machine, in summary, embodies all of the ele- 


~ ments of the controlled operant, and the literature on 


this kind of experiment may therefore be relevant. 

In final summary, the original question is re-asked, 
where should the constructor and programmer of human 
training devices look in the scientific literature of psy- 
chology for experimental data relevant to his task. Any 
answer to this question implies a theory of training de- 
vices. If Skinner’s theory says that the literature on free 
operant is most relevant, this paper proposes the counter 
theory that the literature of controlled operant and classi- 
cal conditioning is also relevant but more so. 

If Skinner’s theory says that memory drum experi- 
inents provide an inappropriate model for construction 
of training devices, this paper proposes the counter theory 
that no other experimental device or procedure more 
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closely resembles Skinner's Writein Machine than a ' 


common memory drum paradigm. 
Old Ebbinghaus’s shoulders may not be ivy-league, but 
we ought to stand on them anyway. 
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A Review of the Literature on 
Certain Aspects of Automated 
Instruction* 


W. J. Carr 
Temple University 


The primary purpose of this paper is to review the fast- 
growing literature on the design and utility of automatic 
teaching devices. A secondary purpose of the paper is to 
suggest a number of hypotheses, the testing of which 
might further our understanding of the variables which 
influence the effectiveness of automated instruction in 
the classroom. 

The plan of the present paper is as follows: First, there 
will be an attempt to provide a conceptual framework in 
which to place a discussion of automatic teaching de- 
vices. Second, a review will be made of the rather exten- 
sive literature on automated instruction. In this section, 
the writer will be concerned with two types of studies, 


*A report submitted (January 1959) to Dr. J. Charles 
Jones, Director, Susquehanna Valley Program, Bucknell Uni- 
versity. Support for this work was received from a grant from 
the Fund for the Advancement of Education to Bucknell 
University. The report is used here with the permission of the 
author. Similar versions of this paper have appeared in 

. Lumsdaine, A. A., and Glaser, Robert (Ed.), Teaching Ma- 
chines and Programmed Learning, Washington, D.C.: National 
Education Association, 1960, and WADC Technical Report 
59-503, Air Research and Development Command (August 


1959). 
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(a) simple evaluation studies, in which the researcher 
compared the effectiveness of a given teaching device 
with the “standard” form of instruction and (b) para- 
metric studies, in which the researcher investigated one or 
more variables of which the effectiveness of the teaching 
device might be a function. Finally, there will be an 
attempt to summarize some of the current thinking as 
to how research in this area might best proceed and to 
suggest some specific hypotheses. 


SOME PRINCIPLES OF LEARNING 


For almost one hundred years, psychologists have ex- 
pended a considerable portion of their research effort 
toward an understanding of the learning process, chiefly 
by studying the variables which influence the rate of 
learning and forgetting. Out of this effort has come a 
number of reliable principles of learning which can be 
utilized to design an efficient learning situation. 

Gagne (1958) has reviewed the literature on learning, 
on a very broad scale. His paper encompasses almost all 
of the known variables which influence learning and 
retention. Moreover, Gagne has discussed at some length 
the importance of using adequate measures of learning, 
which necessarily includes the problem of transfer from 
the classroom to real life. 

Skinner (1954) and Gilbert (1958) have summarized .. - 
those principles of learning which have led a number of 
workers to consider seriously the development of auto- 
matic teaching devices for use in the classroom. A re- 
phrasing of these principles, couched in non-technical 
terms, might take the following form. Accompanying 
each principle is a brief statement showing how that , 
principle is sometimes violated in the typical classroom. 


1. Learning takes place most rapidly if the student is 
actively engaged with the subject matter. At the outset 
of training, some time might profitably be spent in 

“watching or listening to someone else perform the acts 
to be learned, but the student will become proficient 
only if he practices the acts himself. This principle ap- 
plies to verbal skills as well as to non-verbal ones. Unfor- 
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tunately, in most classrooms the learning of most subject 
matter is thought of as being a spectator, rather than a 
participator sport. 

2. Learning is most effective if the student develops 
the skills and knowledge in a form which will readily 
generalize to the “real life” situation for which they are 
intended. Usually, this means that the student must learn 
to construct correct answers to questions, rather than 
merely be able to recognize them. Thus, the responses 
which the student makes in the learning situation should 
be of the former rather than of the latter type. All too 
frequently, in the classroom of today, recognition is used 
as a criterion of learning. 

3. Learning takes place most rapidly if immediate 
“knowledge of results” 1 is given for each response. If the 
response is correct, the student should be so informed, 
just as soon as the response is completed. If the response 
vis incorrect, he should know this, also. Even a slight delay 
drastically retards learning. Presently, our students must 
frequently endure long delays before they receive feed- 
back of this sort. 

4. Learning takes place most rapidly if the subject 

_ matter organized in a hierarchic form. Lessons should 
begin with very simple problems, for which the student 
already has the necessary skills and knowledge. Gradually, 
the difficulty level may be raised until the desired level 
of proficiency is obtained. Care must be exercised so that 
at each point the student is given new information to 
be used to solve the next problem. Ideally, the subject 
matter should be presented in steps so small that the 
success of the student on the next problem is practically 
guaranteed. Unfortunately, both textbooks and classroom 
techniques frequently ignore this principle. 

5. Receiving frequent “knowledge of results’ keeps 
students working at the assigned task. The hurdles which 
the student must surmount should be kept immediately 


1The sophisticated reader will recognize that mere “knowl- 
edge of results” does not guarantee rapid learning. Strictly 
speaking, learning takes place only if the “knowledge of re- 
sults” also constitutes reinforcement. A reinforcement is a 
stimulus which increases the probability of the stimulus- 
response connection which it follows. 
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before him and by making successes frequent, the student 
will develop a high level of aspiration. In the typical 
classroom, the student is studying to prepare himself for 
a far-off quiz or test. The result is that the teacher must 
spend a considerable portion of her time devising some 
artificial motivators, to keep the students working. 

6. Since learning takes place in individuals, the learn- 
ing situation should be designed so that each student 
may proceed at his own pace. For various Teasons, some 
students learn faster than others and therefore, it is diffi- 
cult to have them learn in groups.? This principle does 
not necessarily dictate that students must work alone. It 
merely emphasizes the belief that if students are to learn 
in groups, the learning situation should be designed so 
that the fast learners are not impeded by the slow ones 
and the slow learners are not confused by too stiff a pace. 


The list of principles given above is neither new nor 
exhaustive. Good teachers have known of theses princi- 
ples for a long time and have endeavored to utilize them 
as best they can, within the limitations of the present-day 
classroom situation. 

But it must be recognized that the limitations imposed 
on the teacher by the present-day classroom situation are 
extremely severe. Many educators and psychologists have 
been impressed with the folly of trying to teach thirty or 
more children, simultaneously. Consider, for example, the 
teaching of the multiplication tables. While one student 
recites, the teacher is able to reinforce each of his re- 
sponses. But the other children are merely listening and 
probably little learning occurs. Further, if the teacher 
uses the weekly quiz as a learning device, learning suffers 
from the long delay between the responses and the 
knowledge of results. Finally, if the class is working on 
the “five” tables, some bright students might more 
profitably be working on the “sevens” while some of the 
slower ones should still be working on the “threes.” True, 
making the class more homogeneous with respect to 
mathematical ability might help, but this poses special 
administrative problems. 

2 The writer recognizes that the development of some skills 


requires the Presence of a group. In this discussion we have 
teference to those skills and knowledge which do not. 
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The teacher's inability to work efficiently with more 
than a very few studetns at a time has caused a number 
of psychologists to follow the lead of Sidney L. Pressey 
and B. F. Skinner in recommending a thorough inves- 
tigation of teaching by means of automatic teaching 
devices, Presumably, each student would have the daily 
use of a machine for a period of time. Theoretically, at 
least, the machine could be designed so that its opera- 
tion would be congruent with all of the principles of 
efficient learning which have been here reviewed. 

Skinner (1958) has discussed some of the social and 
economic ramifications of his proposal to make wide use 
of automatic teaching devices in our educational system. 
Space will not be devoted to this topic in the present 
paper, except to point out that few if any workers in this 
field intend to build devices designed to replace the 
teacher in the classroom. At the very best, the devices can 
only hope to supplement the efforts of the teacher, by 
relieving her of the task at which she is probably least 
capable—delivering “knowledge of results” rapidly to her 
students for each response directed toward the subject 
matter. If effective, the use of such devices would free the 
teacher for tasks which she can perform well and for 
which no machine can replace her. 


A REVIEW OF LITERATURE 


Porter (1958a) has recently published an excellent 
review of a portion of the extensive literature dealing 
with the many devices which have been suggested as 
being useful adjuncts to the teaching process. Further, 
he has developed a system of classification of these de- 
vices which promises to be quite useful. He has distin- 
guished among three classes of devices: 

Stimulus Devices: Stimulus devices are those which 
present some information to the learner, by way of one 
or more sense modalities. Frequently, they show how or 
under what conditions something is to be done. Exam- 
ples of this class of device are: models, projecting devices, 
phonographs, braille, and the like. It should be pointed 
out that while these devices do present subject matter to 
the learner, they do not necessarily require any action on 
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his part and thus learning is not assured. For this reason, 
Porter suggests that they be thought of as teaching aids, 
rather than teaching devices. 


Response Devices: Response devices are those which 
provide the learner with an opportunity to practice some 
activity. These devices are usually used to collect or ma- 
nipulate data. Examples of this class of device are type- 
writers and desk calculators (as data manipulators) and 
the Classroom Communicator or the GSR recorder (as 
data collectors). Thus, a response device permits the 
learner to practice some activity, but it does not neces- 
sarily provide him with subject matter on which to prac- 
tice. For this reason, Porter suggests that response devices 
also be thought of as teaching aids, rather than as teach- 
ing devices. 

Stimulus-Response Devices: Stimulus-Response devices 
are those which both present information to the learner 
and also require some appropriate action on his part. 
Further, such devices usually respond differentially, de- 
pending upon the behavior of the learner. Examples of 
the stimulus-response class of device are the Pressey 
multiple-choice apparatus, Skinner’s devices and the clas- 
sical memory drum. It should be noted that the stimulus- 
response device more nearly simulates the role of the 
teacher by both presenting information to the learner 
and requiring some appropriate action on his part. Porter 


feels that these might properly be called teaching de- ` 


vices, since no human teacher is required to mediate 
relations between the machine and the learner, except 
to service or maintain the machine. 


Further, Porter distinguished among three types of 
stimulus-response devices. First, there is the simulator, 
which duplicates the essential characteristics of some 
complex task and which requires appropriate action from 
time to time on the part of the learner. Ordinarily, this 
type of device does not provide immediate feedback. The 
electronic flignt simulator is a good illustration of this 
type of device. 

The second type of stimulus-response device distin- 
guished by Porter is the immediate reinforcer, which not 
only presents the learner with a problem situation but 
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also provides him with immediate “knowledge of results” 
concerning his answers. It should also be noted that this 
type of device is learner-paced, in that the machine’s 
reaction is contingent upon the learner's behavior. Pres- 
sey’s multiple-choice device and Skinner’s teaching ma- 
chine are examples of the immediate reinforcer. 

The third type of stimulus-response device is what 
Porter called the pacer. A pacer is a device which pre- 
sents stimulus materials for a given interval of time and 
then provides the appropriate response, whether or not 
the learner has attempted to answer. The memory drum, 
so frequently used in psychological research on learning 
phenomena, illustrates this type of device. 

Of the three types of stimulus-response devices distin- 

guished above, the one which seems to offer the most 
promise for use in the class is the immediate reinforcer. 
Simulators ordinarily do not provide immediate feedback 
and the operation of the pacer is independent of the 
behavior of the learner. Thus, these latter types of de- 
vices seem to violate one or more of the principles of 
learning referred to earlier. Hereafter, in the present pa- 
per, consideration will be given only to those stimulus- 
response devices which are of the immediate reinforcer 
type. 
a word of caution might be added with respect to 
Porter's classification of learning machines. It is quite 
possible that a slight modification of a stimulus-device or 
a response-device might enable it to qualify as a stimulus- 
response device. Also, it is possible for both simulators 
and pacers to be reclassified as immediate reinforcers, if 
appropriate changes are introduced. Porter has classified 
particular devices in accordance with the design of the 
machine offered by its designer. 


SOME HISTORY OF AUTOMATED INSTRUCTION 


Perhaps the first attempt to develop an automatic 
teaching device was made in 1918 by English (1942). 
This worker constructed and tested a device by which 
soldiers rapidly learned to squeeze rather than jerk a 
rifle trigger. Knowledge of results was delivered to the 
soldier via a manometer attached to the trigger. However, 
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it should be pointed out that the subject matter in this 
case was quite simple, consisting of a single habit. 

Sidney Pressey (1950) is often credited with being the 
first to give serious thought to the widespread use of 
teaching devices in the classroom. Although, at the outset, 
Pressey was primarily interested in automatic testing 
devices, he soon discovered that students could learn b 
taking tests via his machines. Pressey and his students 
have published a good number of papers in which various 
designs of machine have been described and evaluated. 

Basically, the type of device developed by Pressey 
follows the format of the multiple-choice question. The 
student is presented with a question and four options, 
only one of which is correct. He chooses an option and 
the device provides some immediate feedback. If the 
student has chosen the correct answer, the machine 
presents the next question. If an incorrect option is 
chosen, the student must continue responding until he 
selects the correct option. 

Three different recording devices were described by 
Pressey and his students. They developed a punch-board 
device, a system using chemically treated paper, and an 
electrical system. See Pressey (1950) for a more complete 
description of each. 

The type of device developed by Pressey suffers from at 
least two defects which seem quite important. First, the 
student must merely recognize the correct answer among 
a number of options. In many cases this format is quite 
unlike the real life situation for which the skills are being 
developed and the problem of transfer arises.3 Second, 
the materials which Pressey presented were not arranged 
in any hierarchic fashion. The lack of carefully prepared 
sequence of information does violence to one of the 
principles of learning described earlier in this paper. 

Recently, Skinner (1958b) developed a different type 
of teaching device which circumvents the limitations of 
the Pressey machines. Using the devices developed by 
Skinner, the learner must construct, rather than merel 
recognize correct answers to questions. Further, Skinner 
and his colleagues spent a considerable portion of their 


*Kendler (1958) has discussed some of the i 
of transfer connected with teaching devices. arena ouster 
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research time, in attempting to construct programs of 
subject matter, beginning with simple problems and grad- 
ually increasing complexity only as the learner demon- 
strates that he is ready for it. For a good illustration 
of i is meant by a program, see Skinner (1958a, pp. 
-7). 

One of Skinner’s machines is used to teach arithmetic. 
The learner constructs his answer by operating sliders, 
each of which represents a digit. In this way, the learner 
can construct answers up to four digits in length. For 
materials other than arithmetic, the learner writes out 
his answer. Then he may compare his answer with the 
correct one, merely by operating a lever. The reader can 
easily see that such a procedure introduces a source of 
error, since the learner decides whether or not his answer 
is correct. In some models, the machine provides for 
recycling of those questions which the learner had wrong 
cn previous trials. 


PARAMETERS INFLUENCING THE EFFECTIVENESS 
OF AUTOMATED INSTRUCTION 


A good deal of the research on automated instruction 


“has taken the form of evaluation studies, in which teach- 


ing by machine has been compared with teaching by the 
“usual” method. The present writer shares with Gilbert 
(1958, p. 29) the belief that while a certain amount of 
evaluation research is necessary in order to justify a 
continued interest in the basic concept of automated in- 
struction, the major portion of the research effort should 
be devoted to an experimental analysis of the parameters 
which influence the effectiveness of automated instruc- 
tion. This latter type of study will be discussed first. 

A distinction may be drawn among three classes of 
variables of which the effectiveness of automated instruc- 
tion might be a function. They are: characteristics of the 
device, of the program, and of the learner. Each of these 
classes of variables will be discussed in some detail and 
an effort will be made to summarize the pertinent re- 
search findings. 
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Characteristics of the Device: The reader will recall + 
that, earlier in this paper, a teaching device was defined 
as a machine by which stimulus materials are presented 
to a leamer and which requires some response on his part 
in order for the device to operate. Further, the type of 
teaching device with which this paper is primarily con- 
cerned is one which provides immediate feedback to the 
learner. This type of analysis permits one to discuss the 
characteristics of the device within the framework of a 
three-fold analysis: 

1. Display or input characteristics: So many specific 
devices have been used by workers as means of presenting 
the stimulus materials that a complete classification of 
them is almost impossible. Suffice it to say that most 
workers have made use of mechanical, electrical, or optical 
components to present information to the learner. Two 
exceptions may be noted. Crowder (1958b) and Homme 
& Glaser (1958) have successfully presented stimulus 
materials in the form of a modified textbook. And Rath 
& Anderson (1958) have used the IBM “650” print-out. 

To the writer’s knowledge, no experiment has been 
performed which compares the efficiency of different 
methods of presenting stimulus materials. 

2. Response or output characteristics: Two basically 
different modes of responding have been employed. 
Pressey, and his colleagues, and some researchers in 
military establishments have made heavy use of the 
multiple-choice or recognition mode, while Skinner and 
others have used the “free operant” or construction mode 
of responding. No study was found which permits the 
comparison of these two basic modes of operation. 

However, Irion & Briggs (1957) have experimentally 
compared several variations of the multiple-choice mode 
of operation. The four modes compared were (a) quiz 
mode: in which the S chooses one of the Tesponses by 
pressing a switch and the machine immediately indicates 
which of the responses was correct, (b) modified-quiz 
mode: which 1s identical to (a) except that a buzzer 
indicates that an incorrect answer was made, (c) practice 
mode: in which the S must press switches until the correct 
response is made, and (d) single-try mode: in which the 
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S may respond only once and then the next stimulus item 
is presented. 

Each of the four modes of operation was employed by 
different groups of subjects as they learned three types of 
tasks: serial learning, paired-associate learning, and prob- 
lem solving. The dependent variables were the number 
of trials required to learn the tasks to a given criterion 
and a measure of retention, taken two weeks after learn- 
ing. 
In general, the relative efficiency of the four modes of 
operation was in the order: quiz-mode, modified-quiz 
mode, practice mode, and single-try mode. The amount 
of difference in efficiency was a function of the type of 
learning task. Later, Irion & Briggs used different modes 
of operation within a learning situation and found no dif- 
ferences in amount of time to criterion. 

3. Reinforcement characteristics: Little attention has 
been given to the problem of the optimal type and 
schedule of reinforcement to use as the means of main- 
taining the maximal rate of performance. Most workers 
report that sheer “knowledge of results” suffices to keep 
the learner working. Skinner (1958) reports that students 
work harder and longer when using the machine than 
they do when they merely read the textbook. The implica- 
tion is that “knowledge of results” is an effective rein- 
forcer, but it seems clear that various methods of pro- 
viding feedback might very well influence the learner’s 
performance rate. For a more complete discussion of this 
subject, see Porter (1957, pp. 134-136). 


Characteristics of the Program: Programming refers to 
the arrangement of the stimulus materials in the order 
of presentation which will maximize the rate of acquisi- 
tion of the habits to be learned. Presumably, proper 
programming permits the gradual shaping of the learner’s 
responses from low to high levels of approximation of 
the desired responses. 

It should be noted that Pressey and his group gave 
little attention to the problem of programming, perhaps 
because they were primarily interested in developing 
testing devices and only secondarily interested in develop- 
ing teaching devices. Further, it is likely that some types 
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of subject matter require little in the way of programming 
at all (e.g. trouble shooting and the identification of 
objects such as enemy aircraft). 

But for certain kinds of complex tasks, it is likely that 
efficient learning can take place only if the materials are 
carefully programmed into a hierarchy, extending from 
very simple to complex habits. See Gilbert (1958) for 
a more complete discussion of this point. 

Several problems associated with programming im- 
mediately present themselves. 

1. Extrinsic Versus Intrinsic Programming: Of those 
workers interested in teaching devices, Skinner was the 
first to give serious attention to the matter of program- 
ming. For a given subject matter, he attempted to con- 
struct a single program, through which all learners would 
proceed. This type might be thought of as a straight-line 
program. Later, Crowder (1958a, 1958b) described an- 
other type of programming which tailors the program to 
the individual learner, depending upon his strengths and 
weaknesses (as reflected by his previous errors). Crowder’s 
might be thought of as a branching program. The 
branches would permit the learner to tetrace his steps 
back through that portion of the program which his 
errors indicate that he did not adequately learn. Crowder 
referred to his type of program as intrinsic and he re- 
served the term extrinsic Program for the type described 
by Skinner. 

For obvious reasons, the intrinsic Program would place 
a special burden on both the author of the program and 
on the teaching device used. However, the use of an 
intrinsic program might very well result in a more efficient 
learning situation. Unfortunately, no experiment has yet 
been done which compares the two methods of program- 
ming directly. 

2. Size of Step: Skinner (1958) and others feel that 
the most effective program is one which raises the level 
of the learner's proficiency by such small steps that the 
probability of the learner being Wrong is so low as to be 
negligible. In fact, Skinner uses the Percentage correct 
as a measure of the optimum size of step. 

_ On the other hand, some have argued that the optimal 
size of step is the one which results in the most tapid 
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` learning to a given criterion and that this optimum can 
only be determined experimentally. Homme & Glaser 
(1958) investigated the effect of size of step upon fre- 
quency of errors and upon immediate and delayed reten- 
tion. Using a single program on elementary number 
theory, they varied the number of steps over four values: 
30, 40, 51, and 67. Presumably, the greater the number 
of steps, the smaller the mean size of step. Independent 
groups of five Ss each were used. The results showed that 
the smaller the step, the fewer the errors and the greater 
the retention. However, it is conceivable that increasing 
the number (and therefore decreasing the size) of steps 
beyond 67 might have resulted in no further change in 
dependent variable performance. Further, the relation- 
ship between measures of learning and size of step might 
interact with characteristics of the program employed. 
Clearly, further research is necessary before this important 
question can be answered unequivocally, but a start has 
been made. 

3. Rules or Principles of Programming: Skinner 
(1958), Beck (1958), and Gilbert (1958) have de- 
scribed a number of rules or principles of programming. 
Although these rules or principles seem eminently logical, 
they appear to lack empirical support. Thus, they might 
better be thought of as hypotheses to be tested. Space 
does not permit a detailed discussion of these tules and 
the interested reader is referred to the appropriate 
references. 


Characteristics of the Learner: A third major variable 
of which the effectiveness of automated instruction might 
be a function has to do with some characteristics of the 
learner. For example, one might hypothesize that the use 
of teaching devices might wipe out differences in achieve- 
ment measures associated with intelligence or aptitude 
test performance. The findings of a number of experi- 
ments seem to support this hypothesis. 

Porter (1958) found that the correlation between IQ 
and achievement in spelling was lower for a group of 
learners who were taught by machine than for a control 
group who were taught in the usual fashion. Ferster & 
Sapon (1958) reported similar findings on the relations 
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between aptitude and achievement in a course in Ger- 
man. Finally, Lumsdaine (1958) showed that the ad- 
vantage of active participation in a simple learning task 
decreases with increases in intelligence. 

The experiments just cited might lead one to believe 
that superior students do not learn efficiently by machine. 
Pressey (1950), Briggs (1947) and Jensen (1949) have 
all reported data which show that bright students are 
quite capable of benefiting from the use of teaching 
devices. 

Another characteristic of the learner which might in- 
fluence the effectiveness of automated instruction is the 
level of anxiety of the learner with respect to the subject 
matter being taught. Porter (in a personal communica- 
tion) has suggested that learners who are quite anxious 
or who have a low level of aspiration concerning the 
subject matter seem to be ones who profit a great deal 
from machine instruction. It seems reasonable to suppose 
that since the machine instruction provides for many 
reinforcements, the learner’s degree of anxiety might be 
teduced and his level of aspiration raised. The result 
should be an increase in rate of learning, but this hypothe- 
sis has yet to be tested. 


THE EVALUATION OF AUTOMATED INSTRUCTION 


A good number of studies have been located which 
can be used to compare the effectiveness of teaching by 
machine with teaching by the “usual” classroom tech- 
niques. A number of these studies were performed using 
one or another variation of Pressey’s multiple-choice type 
of device. All studies conclude that there is something 
to be gained by using machine type of instruction. Usu- 
ally, the advantage is expressed in terms of the superior 
performance of an experimental group (those using teach- 
ing devices) over a control group. 

Porter (1957, pp. 137-40) has carefully reviewed most 
of these studies and has pointed to a number of defects. 
In many of the studies cited, little if any effort was made 
to equate the groups for initial proficiency with Tespect 
to the subject matter, for level of motivation, for the 
“novelty” effect, or for the amount of time spent in 
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study. Further, although immediate reinforcer teaching 
devices were used in all of these studies, it cannot be said 
that these studies demonstrate the superiority of im- 
mediate versus delayed reinforcement since it seems clear 
that the control groups received less reinforcement as 
well as delayed reinforcement. 

Several evaluation studies have been performed using 
Skinner’s type of device in which the learner constructs 
his answer (rather than merely identifies it) and which 
presents the stimulus materials in a carefully arranged 
hierarchic sequence. Porter (1958) has shown that the 
use of such a device and program for the teaching of 
spelling in second and sixth grade students renders them 
superior in spelling achievement to a control group. 

Homme & Glaser (1958) recently reported a study 
in which comparisons were made between groups who 
learned by using programmed textbooks and groups who 
used standard textbooks. Two types of subject matter 
were used: sections of a statistics course and a course in 
the fundamentals of music reading. In both cases, the 
experimental groups outperformed the control groups on 
achievement tests. No mention is made in this study of 
the procedures used to equate groups for initial pro- 
ficiency, amount of time spent in study, and so on. 

Holland (1958) and Ferster & Sapon (1958) have 
also reported the results of automated instruction, using 
college students as subjects. Holland had students learn 
sections of a course in psychology and Ferster & Sapon 
used the subject matter of a course in German. Control 
group data were not reported. 

On the basis of this cursory review of the validity 
studies, one might conclude that we are still lacking in 
carefully executed experiments which unequivocally dem- 
onstrate the superiority of automated instruction over the 
“usual” classroom procedures. And too, one might say 
that we know even less about why automated instruction 
might prove to be superior. It is entirely possible that all 
of the validity studies are dealing with a “novelty” effect. 
However, Porter (1958) has shown in his year long 
study of spelling achievement that the second semester's 
performance of the experimental group was just as 
superior to the control group as was the first semester’s 
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performance. Such evidence would suggest that if there 
is a “novelty” effect operating in Porter's study, it is slow 
in disappearing. Nevertheless, it is very clear that a great 
deal of additional research is in order before we can be at 
all confident about the superiority of automated instruc- 
tion over the “usual” classroom procedure, 


A FEW HYPOTHESES TO TEST 


The review of the literature presented in the previous 
section of this paper provides a number of hypotheses 
which seem worth testing. This discussion of hypotheses 
might well be organized around the three major classes 
of variables of which the effectiveness of automated 
instruction is a function. 


Characteristics of the Device: The writer shares the 
view expressed by Gilbert (1958a) that, for the present, 
most of the research on automated instruction might best 
be spent by studying relations between characteristics of 
the program and rate of learning or amount of retention. 
However, at least three characteristics of the teaching 
device might profitably be examined immediately. 
Hypornesis 1. The use of self-paced instructional de- 
vices yields more rapid learning and better retention than 
does group-paced devices and that this relationship holds 
for all types of learning tasks and for all types of learners, 

Lumsdaine (1958) has suggested that group-paced de- 
vices might prove to be almost as effective as those which 
permit the individual learner to proceed at his own pace. 
Although there is good rationale for the belief that in 
most cases the self-paced device will prove to be the 
more effective of the two, this is still an empirical 
question. Further, the advantage of the self-paced device 
over the group-paced type might interact with both the 
nature of the task to be learned and with a number of 
characteristics. of the learner. Finally, the advantage of 


Hyrornesis 2. The use of reinforcers in addition to 
knowledge of results” will result in a higher rate of 
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performance on teaching devices and perhaps render 
“knowledge of results” (as a reinforcement) more resist- 
ant to satiation. 

As mentioned earlier in this paper, most researchers 
seem to believe that mere “knowledge of results” is 
sufficient reinforcement to keep students working. How- 
ever, it is entirely possible that prolonged exposure to 
such reinforcement may result in an effect akin to stimu- 
lus satiation. Porter (1957) has warned of just such a 
“novelty” effect. 

Moreover, it may well be that the use of other rein- 
forcers in addition to “knowledge of results” might result 
in an increase in rate of performance. Skinner (1954, 
pp. 93-94) has suggested that approval from the teacher, 
being permitted to engage in desired activities, successful 
competition with other students, and even aversive stimu- 
lation might be used to augment the reinforcing properties 
of “knowledge of results.” 


Hyporuesis 3. There is an indirect relationship between 
the amount of delay of “knowledge of results” and the 
rate of learning and this relationship holds for all types 
of subject matter and for all types of learner. 


Pressey (1950, p. 418), in particular, has argued that 
the superiority of teaching devices over the more standard 
pedagogic practices stems from the fact that the devices 
minimize the delay between the learner's responses and 
“knowledge of results.” But Porter (1957, p. 138) has 
pointed out that in several of Pressey’s validating studies, 
the amount of “knowledge of results” varied simultane- 
ously with the delay in “knowledge of results.” Thus, we 
are left without knowing which of the two factors is the 
more important. Deese (1958, p. 199) reports a study by 
Saltzman which shows that a delay in “knowledge of 
results” can seriously hamper the learning of a rote 
verbal task. A delay of but six seconds resulted in a 50 
per cent increase in errors. - 

However, it may well be that certain types of subject 
matter and perhaps certain types of learner might be 
able to withstand fairly long delays without affecting 
learning rate appreciably. Such information would be of 
some help to the designers of teaching devices. 
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Hyrornesis 4. Having students learn to compose cor- 
rect answers to questions results in faster learning, better 
retention, and more positive transfer to “real life” situa- 
tions than does having students learn to recognize correct 
answers from among a number of options. Further, this 
relationship holds for all types of learning situations and 
for all types of learners. 

Skinner (1954) contends that, for most learning 
situations, it is more desirable to render learners capable 
of composing correct answers to questions, rather than 
make them able merely to recognize them. There are 
good reasons for believing that Skinner is correct on this 
point, but once again, this is an empirical question. Fur- 
ther, the advantage of composition over recognition may 


interact with the type of task and with characteristics 
of the learner. 


Characteristics of the Program: As mentioned earlier 
in this paper, the writer believes that the usefulness of 
teaching devices is more a function of characteristics of 
the program than a function of characteristics of the 
device, itself. For the present, many believe that program 
writing is an art, which is perhaps another way of saying 
that we know little about the characteristics of the op- 
timal program. 

However, some program characteristics have already 


received some study. Crowder (1958b) has distin- | 


guished between extrinsic and intrinsic program tech- 
niques. Homme & Glaser (1958) investigated the 
characteristic called “size of step.” Further, Beck (1958) 
and Gilbert (1958) have both written at length on a few 
principles of programming. Perhaps these principles might 
be put into the form of hypotheses worthy of experi- 
mental test. For these reasons, the following hypotheses 
are suggested: 


Hyroruesis 5. Intrinsic programming yields more rapid 
learning and better retention than does extrinsic program- 
ming. Further, this relationship holds for all types of 
subject matter and all types of learner. 

Once again, the reader should keep in mind the fact 
that although intrinsic Programs might prove superior 
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` to extrinsic ones, the intrinsic type places a considerable 


burden upon both the program writer and upon the 
teaching device, itself. One way of reducing the mechan- 
ical complexity of the device would be to use a pro- 
grammed textbook, intrinsically organized. Crowder 
(1958a, 1958b) has already put this idea to use. 


Hyrornesis 6. The rate of learning via automated in- 
struction is a function of the size of step employed in the 
program. Further, if properly measured, the optimal size 
of step is a constant for all types of subject matter. 


When a student is taught a given subject matter, it 
is assumed that he starts off at a given level of proficiency 
and, after training, ends at a higher level. Now, one may 
attempt to go from the lower to the higher level by 
presenting the learner with relatively few or many steps. 
Further, assuming that the steps are roughly equidistant, 
then size of step is negatively correlated with the number 
of steps. Thus, both number of steps and size of step 
might be variables of which the rate of learning is a 
function. 

Skinner (1958, p. 5) feels that steps should be so 
small that errors are rarely if ever committed. In fact, 
some workers feel that the percentage of errors is a meas- 
ure of the worth of an item. Items which result in more 
than a minimum number of errors are viewed with 
suspicion. On the other hand, some workers feel that 
some percentage of error might well be built into the 
program and presumably this would be done by increas- 
ing the size of step in order to obtain the desired per- 
centage of error. 

The question of the optimum size or number of steps 
in a given program is obviously an empirical one, to be 
answered by experiment. Homme & Glaser (1958) have 
shown that increasing the number of steps (and thereby 
decreasing their size) resulted in more efficient learning. 
However, it is entirely possible that Homme & Glaser did 
not vary the number of steps in the program in an asymp- 
totic function or even in a decrease in learning efficiency. 
Clearly, this variable is worthy of careful examination. 

Finally, the writer believes that subject matter cur- 
rently thought of as being difficult may simply be ma- 
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terial which is programmed + in very large steps. If a 
technique could be developed which would permit the 
measurement of size of step in units which provide an 
equal-ratio scale, then one could test the hypothesis that 
the optimal size of step is a constant for all types of 
subject matter. Unfortunately, the writer knows of no 
such technique. 


Characteristics of the Learner: As mentioned earlier, 
several studies have shown that the amount of improve- 
ment shown by students is relatively uncorrelated with 
their intelligence and/or aptitude for the subject matter. 
Such data must be interpreted carefully. They do not 
imply that students with low intelligence or aptitude 
profit more from automated instruction than do “bright” 
students. Rather, these findings merely state that the 
amount of profit is independent of such factors as intel- 
ligence and aptitude. 

Researchers would do well to continue to search for 
characteristics of the learner which are correlated with 
amount of improvement. For such information would 
probably give us hints about the optimal design of the 
program, the type and schedule of reinforcement, and 
of the teaching device, itself. 


Hyporuesis 7. The rate of learning via automated in- 
struction is a function of level of anxiety and/or level 
of aspiration of the learner with respect to the subject 
matter, 

Porter® has noted clinically that one benefit of auto- 
mated instruction seems to stem from the fact that some 
learners become less anxious about their performance on 
the subject matter after they have used a teaching device 
for some time. Such an observation suggests that im- 
mediate knowledge of results and perhaps a high per- 
centage of correct responses might be more beneficial to 
learners who are anxious than to those who are not. 


“The reader “shouid recognize that teachers and textbooks 
Program material, in the sense that the material is arranged 
ma hierarchy conducive to rapid learning. 

Personal communication. 
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NOTES ON DIFFICULTIES INHERENT IN RESEARCH 
ON AUTOMATED INSTRUCTION 


Porter (1957, pp. 137-38) and Gilbert (1958, pp- 
29-32) have discussed some of the difficulties inherent in 
research on automated instruction. Both workers feel 
that most evaluation studies presently available simply 
do not meet the necessary criteria of excellence which 
permit us to state categorically that automated instruc- 
tion results in performance superior to instruction via 
the “usual” techniques. Most studies simply lack suf- 
ficient experimental control. The experimental and con- 
trol groups differ in so many respects that it is impos- 
sible to tell which of the differences in treatment account 
for the differences in performance. 

The chief difficulty seems to stem from the inability to 
control the subjects” previous experience with, and apti- 
tude for, the subject matter to be learned. Most studies 
have used regular students as subjects in classrooms at 
all levels of education. Further, the subject matter em- 
ployed is usually standard material. Finally, many of the 
studies have employed such large numbers of subjects 
that it is practically impossible to learn much about what 
is taking place while an individual is learning. 

The writer suggests that at least some research time 
might be devoted to the careful observation of small 
numbers of subjects while they are learning materials with 
which they are totally unfamiliar. The classical non- 
sense syllable or objects frequently used in concept forma- 
tion studies recommend themselves. Zeaman (1958) re- 
cently discussed the use of the latter type of material 
within the context of automated instruction. 

If, for one reason Or another, the researcher desires to 
do research in a real classroom, he might give thought to 
the teaching of relatively discrete skills which are su 
ject matter for only one or two years of schooling. For 
example, at the elementary school level ore might easily 
program the teaching of “telling time” or the use of 
roman numerals. Such instruction rarely interacts with 


other types of subject matter. At the collegiate level, the 


teaching of parliamentary rules might be more easily 


78 A Review OF THE LITERATURE 


programmed than the teaching of algebra, philosophy, 
and the like. 

Further, ease of programming is not the only benefit 
which might obtain from the use of subject matter de- 
scribed above. Since such material is relatively discrete, 
the techniques used to teach it will not interact with 
the learning of other subject matter. 
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Teaching Machine Dichotomy: 
Skinner vs. Pressey* 


Epwarp B. Fry 
Loyola University, Los Angeles 


Even though the field of teaching machines is not very 
} old or very large, there has already developed what ap- 
_, pears to be a relatively major dichotomy. On one hand 
ASD we have the historically older faction headed by Sidney 
= Pressey at Ohio State University, which emphasizes 
| multiple-choice devices; and on the other, a newer fac- 
_ tion headed by B. F. Skinner at Harvard University, 
which emphasizes construction of response devices. Nu- 
merous theoretical and practical considerations (amount 
of error permitted and methods of programming) enter 


/ t 
eo into the dichotomy. 

The common characteristics of teaching machines may 
be described simply. These are devices which tutor the 
student without assistance from a human instructor. They 
require the student’s active participation. He must re- 
spond to questions which may be interspersed with 
graphic or verbal subject matter, or with other techniques 
as the Socratic method or “vanishing” (gradually remov- 

| ing part of a word or phrase until the student responds 
A, 
me 


* Paper presented at the annual convention of the American 
Psychological Association, Cincinnati, September 1959. This 

> article originally appeared in Psychological Reports, 1960, 6, 
ieir 11-14, and is used here with the permission of the author and 
the editor. 
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without a cue). Two basic principles are involved: (a) 
complete variability in rate, rate being determined by the 
student’s responses which permit individual differences, 
and (b) immediate knowledge of results, i.e., the device 
“rewards” the student with the answer immediately after 
he responds. 


TYPES OF QUESTIONS AND LEARNING THEORY 


The first major difference between the two schools is 
the type of questions preferred. Pressey prefers the mul- 
tiple-choice type of question, partly because originally 
he envisioned his teaching machine as also an automatic 
testing device. He states (1926) that permitting the 
student to make more than one error per question (his 
“degree of ignorance” per question) greatly increases the 
spread and significance of the total score. Learning 
theory at that time was strongly influenced by association- 
ism. Pressey referred to the laws of recency, frequency, 
and effect, but was also willing to use questions which 
called for “insight.” 

Skinner prefers that the student construct his answers 
rather than relying upon recognition. On his arithmetic 
machine (1954) the student responds by moving digit- 
marked sliders to indicate the correct answer. The con- 
structed-response type of question is in line with Skinner’s 
emphasis on the principle of operant conditioning in 
which characteristic learning is controlled by rewarding 
the student after he has made the correct response. 
Skinner attempts to structure the situation so that re- 
sponses lead step by step toward the desired goal, thus 
shaping behavior through successive approximations. 


AMOUNT OF ERROR DESIRABLE 


Skinner would avoid the structuring of wrong answers 
in programming questions for his teaching machine. 
“Why cause ary learning at all between the wrong answer 
and the right question?” he asks. He would be happy if 
every student got every question correct. 

Pressey, on the other hand, is willing to allow some 
error. Although specific amounts of error are seldom 
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mentioned, Pressey would say (and most psychologists 
would agree) that the number of right responses should 
greatly exceed the number of wrong. 


SUPPLEMENT OR REPLACE OTHER 
TEACHING METHODS? 


Although perhaps a debatable point, the dichotomy 
extends to the instructional uses planned for the two 
types of machines, Pressey’s devices having been used. 
more as supplemental adjuncts to courses and textbooks, 
and Skinner's as replacement for texts and classroom in- 
struction. For example, Porter (1958) used a simple 
write-in machine to supplant spelling instruction; ‘No 
spoken instruction was given.” By contrast, Pressey 
(1950) used multiple-choice questions with self-scoring 
punchboards to supplement regular instruction in educa- 
tional psychology. [Incidentally, results of both studies 
favored the experimental groups. Neither Pressey nor 
Skinner would eliminate teachers, both being interested 
in making more efficient use of the teacher's time in 
class.] 


IS INTELLIGENCE STRICTLY QUANTITATIVE? 


To Skinner, the difference between a bright and a dull 
student is largely one of quantity of learning. If teaching 
steps are small and understandable, a poor student can 
learn the same thing as a bright student, he just has to 
work longer. And when students find concepts difficult, 
Skinner believes (1958), the instructor has not presented 
the steps in small enough or correctly ordered units. In 
planning curriculum, the programmer must discover and 
properly order the units of information, or steps. 

While Pressey does not deny the truth of Skinner's 
quantitative concepts of intelligence and learning, his 
experiments are structured as though “superior” students 
were, indeed, different (Briggs, 1949; jensen, 1949). 
Crowder (1958) actually presents more and different 
steps for poorer students. Pressey places no stress on the 
smallness of the learning steps or the ordering of presen- 


tation. 
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PRACTICAL APPLICATIONS 


While both sides of the dichotomy envision wide use _ 
for their devices, actual hardware and experimental studies 
are rather scarce. Pressey showed an actual multiple- 
choice testing and teaching machine at the 1924 Ameri- 
can Psychological Association meeting. And, from time 
to time variations have appeared, such as Peterson’s 
chemo-card in 1931 and Angell’s punchboard study 
(1949) in which 81 matched pairs of chemistry students 
used punchboards on regular quizzes. Angell and Troyer 
later developed a commercial punchboard which Science 


study of over 1000 students using punchboards led to 
the production of the punchboard for use in the Navy 
(Hill, 1955). 

Besnard, Briggs, and Walker (1955) ‘developed for the 
Air Force several large multiple-choice devices for trouble- 
shooting. Electronics trouble-shooting research has pro- 
duced numerous other multiple-choice devices, such as 
the Tab Test (Glaser, 1954), Trainer Tester, and Auto- 
mat. Also, Skinner has a small battery of disc-type write-in 
machines which were used by Holland (1958). The 
Navy Special Devices Center has recently constructed a 
small battery of similar machines. Skinner has a slider- 
type of arithmetic-spelling machine produced by IBM. 
Also at Harvard, Porter has several of the very simple 
write-in machines mentioned earlier. 


DICHOTOMY IN NON-MECHANICAL DEVICES 


A new development is not to use machines at all, but 
to use books and paper devices. Although lacking some 
of the control of machines, these devices provide flexi- 
bility of rate, unit presentation, and immediate knowl 
edge of results. Yet non-mechanical devices fall into the 
basic dichotomy discussed here. An example of a new 
multiple-choice paper device is Crowder’s (1958) scram- 
bled book, in which a bit of information is followed by 
a multiple-choice question. The student answers the 
question by turning to the appropriate page; the answer 
page tells whether the choice was correct, and why. If the 


Research Associates published for a time. Pressey’s 1950 
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answer was wrong, further instruction is given, followed 
by a new question. If correct, the next problem is pre- 
sented. . 

Ferster and Sapon (1958) describe a cardboard device 
which requires free response. A cardboard sleeve contains 
a slot which exposes one line of a question on a printed 
paper slip. The student writes his answer on a scrap of 
paper, moves the cardboard down one line to expose the 
answer, scores himself, and proceeds to the next question. 
Another device is that developed by Homme, Glaser, and 
Evans (1958), a “programmed textbook” to which the 
student makes numerous subjective responses. He is given 
the answer on the next page, thus providing him with 
knowledge of results. 


SUMMARY 


We have seen that in the field of teaching machines 
there is a dichotomy of opinion with respect to response 
mode, step size, amount of error desirable, learning 
theory, views on intelligence, and the supplementing of 
traditional instructional methods. What we need now are 
a lot of good experimental studies, both theoretical and 
applied in nature 
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EDITOR'S NOTE 


Three approaches to programming have 
been developed; because of limitations of 
Space only that used most frequently is repre- 
sented in this collection of readings. In David 
Klaus’ article which follows only linear pro- 
gramming utilizing a constructed-response 
format is discussed in detail. This has been 
called the Skinnerian type of program. The 
second approach is called “intrinsic program- 
ming” or branching and it usually is identi- 
fied with Norman Crowder (1959). In this 
technique, the student is given a unit of 
information containing a question or a prob- 
lem followed by a set of multiple-choice an- 
swers from which he selects one. If his choice 
1$ Correct, the student continues on to the 
next unit of information, if his choice is in- 
Correct, he is instructed to go to a sub-pro- 
gram (in a scrambled book, to a page) which 
explains his error to him and instructs him 
to try the same or a similar problem again. 
The third approach combines the linear or 
Sequential technique and the branching tech- 
nique by instructing the student to skip a set 
number of frames if his solution to a prob- 
lem is correct. The difference between the 
Crowder and the Skinner approaches is not 
mere mechanics—It involves a distinct dif- 
ference in philosophy of teaching. A thor- 
ough discussion of these approaches to pro- 
gramming may be found in Rigney and 
Fry (1961). 
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The Art of Auto-Instructional 
Programming 


Davw J. Kraus* 
American Institute for Research 


, With the possible exception of large-scale group test- 
Ing, no one development in the field of psychology seems 
to have as much potential for the better utilization of 
human resources as auto-instructional methods. The tech 
nique has promise for producing a genuine and large scale 
Improvement in educational practices and, as a conie: 
quence, it is expected to have a tremendous impact on t z 
quality of classroom instruction. Because of the novelty S 
the technique and its potential impact on ‘education, 1 
1S not at all’ surprising that auto-instruction 1s mois 
frequently talked about than understood, and that mor 
predictions are made as to its potentials than facts are 
collected as to its capabilities. ; d 

Perhaps more than anyone else, the professional h r 
cator should keep himself well informed as to w He 
auto-instruction has been and where it is going. b t 
should know not only the facts and the forecasts n ut 
since he has the responsibility for wisely using these 
T A 

* Thi icle i er read at the Seminar on 
Heat Reece e a in December 1960. 
"his paper originally appeared in Audio Visual Come 
tion Review, 1961, 9, 130-142, and it is reprinted here wi 
Permission of the author. 
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materials, it is especially important that he become ac- 
quainted with the art of programming—the preparation 
of auto-instructional programs. 

Sooner or later, educators and teachers are going to 
have full responsibility for writing programmed materials 
for classroom use, but unfortunately, programming is 
still a skill about which we know very little. The rules we 
have are incomplete at best, and for this reason, perhaps 
the greatest need at this moment is for a clarification of 
the techniques involved in preparing auto-instructional 
materials. Efforts along these lines will not only facilitate 
increased research and development, but will permit the 
preparation and rapid utilization of a large number of 
programs in our schools. But before illustrating what 
appear to be important characteristics of good as opposed 
to poor programs, it may be helpful to describe a little 
about what programming is, where it has been, and where 
it may be headed. 

In essence, the auto-instructional method promotes the 
orderly and controlled development of an individual’s 
skill in much the same way as a good tutor might do. 
By presenting lessons in small, carefully sequenced steps, 
complicated skills can be developed by gradually pro- 
gressing from very simple to very complex levels of per- 
formance. Since the student must perform actively at 
each step during training, it is possible to carefully guide 
the development of his skill by means of immediate con- 
firmation as. to the correctness of each response. Further- 
more, the auto-instructional technique facilitates the — 
evaluation and improvement of the materials during the — 
course of their development in that the difficulty level _ 
or contribution of each step can be carefully ascertained 
and, when necessary, any step may be modified or — 
revised. | 

In Fig. 1 three frames from physics are used to illus- | 
trate how programmed materials of the kind we have 
been preparing at the American Institute for Research 
appear to the student. As illustrated in Frame l, the | 
student is exposed to material in the form of a small 
step which is designed in such a way as to encourage 
him to respond. Before beginning the program, the stu- 
dent is told to respond while he reads through the frame. 
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In general, he is to write his answers either directly in 
the book where the frame appears, on a separate answer 
pad, or if an auto-instructional device or “teaching ma- 
chine” is used, on a strip of paper. As soon as he has 
completed the three responses called for, the student can 
€xpose the correct answers either by turning a page or 
advancing the machine. There, the student would see 
the three answers, heat, energy, and heat, which he can 
check against the responses he had given. Then, the 


— 
A fire can keep us warm because it gives off energy 
Frame 1 in the form of ..... one The sun also 
gives off . s... in the form 
Of 2... 
Heat is one form Of ..seeeeereeceertet® 
Frame 2 Physicists call it thermal energy. A quart of boiling 
water has more s.......seesseseeeseee energy 
than a quart of cold water. 
r 
As a warm object grows cooler, it loses heat, that is, 
i . When a 
Frame 3 it loses . braasosens easta GNOrgY: e! 
cool object gets warmer, it gains «e+ererrreerrte 
jig that is, it gains* sssssssesrereeeettt et" 


* The asterisk indicates that more than one word belongs in that 
space, 


Fig. 1. Three frames from a program in physics. 


student would again turn the page or advance the pro- 
gram and look at the second frame, shown in Frame Z 
_. The student again would respond and then check to see 
if his answers ate correct. He would then turn to the 
third frame, shown in Frame 3. Learning that heat is 
energy, or more specifically, thermal energy, 1s not very 
astonishing in itself. But in terms of a high school 
physics program of 15,000 or 16,000 frarñes, it can be 
seen how each small step gradually and systematically pro- 
duces 2 substantial amount of learning. F 

Research already accomplished indicates in a general 
way what can be achieved by means of auto-instructiona? 
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programs. Under the sponsorship of the Office of Edu- 
cation, Dr. Lumsdaine and I prepared six weeks of high 
school physics in auto-instructional form. From these 
studies we were able to conclude that when used as a 
supplement to an otherwise full program of instruction, 
programmed materials do add significantly to student 
achievement. Furthermore, the evidence we obtained also 
suggested that auto-instructional programs may be relied 
upon to provide instruction, in physics at least, inde- 
pendent of classroom lectures and recitation. 

The use of auto-instructional programs has not been 
limited, of course, to developing proficiency with the facts 
and concepts of physics. Hoiland and Skinner at Harvard 
University have prepared a considerable portion of the 
introductory psychology course in programmed form. 
Glaser and his associates at Teaching Machines, Incor- 
porated, have prepared a program to teach descriptive and 
inferential statistics at the college level, and Eigen and 
Komoski! at Collegiate School have prepared a program 
to teach logic to children at the elementary school level. 

Industrial organizations have also begun to explore the 
value of auto-instructional materials for their technical 
training programs. We recently assisted a group of psy- 
chologists and instructors at IBM as they began the prep- 
aration of programs to teach the operation and mainte- 
nance of equipment as well as the programming of giant 
computers. At HumRRO in Washington, Rocklyn has 
prepared a program which provides instruction in oral 
Russian which can be used, despite the limited vocabulary 
involved, to deal with Russian speaking individuals; 
Shettel and Angell at the American Institute for Research 
have prepared a program to teach certain perceptual 
skills involved in the operation of a SAGE center to Air 
Force trainees; and Audrey Holland has prepared a 
program at the University of Pittsburgh to provide pre- 
liminary corrective ear training to students with articula- 
tory speech defects. 

We now have reason to believe that even higher order 
skills than these can be taught by means of auto-instruc- 


1Mr. Komoski has been named president of the newly estab; 
lished Center for Programed Instruction, 365 West En 
Avenue, New York, N.Y. 
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tion. We are currently engaged in an investigation of 
methods for teaching art judgment. While the program 
being prepared will include some information on art 
history and principles, it has as its major purpose the 
development of a feeling of appreciation for art and the 
ability to differentiate good art from that which is not. 
We are also attempting to develop a program to provide 
laboratory instruction in electronics which will involve the 
use of actual laboratory equipment and a program to instill 
in students the insight necessary to solve geometric 
theorems which they have never before seen. : 
The success we have achieved thus far in broadening 
the applications of auto-instructional programs has led 
many of us to a feeling of cautious enthusiasm with 
respect to the progress that has been made. As more 
research is completed and more programs developed, we 
should have a still better understanding of the role that 
auto-instructional methods might have in the classroom 
of the future. The research that will contribute toward 
this goal must be carefully done. One of the most encour 
aging characteristics of auto-instructional materials is 
that they almost uniformly are capable of providing in- 
dividual instruction, no matter how well or how poorly 
the program is written. All programs are not, and most 
certainly will not be, of equal quality. Until more ob- 
jective and empirical measures of quto-instructional effec- 
tiveness become available, the educator must be ade- 
quately sophisticated in his judgment to evaluate the 
teaching competence of a program he might use 1n the 
classroom. Í a 
The educator must also be aware of the pitfalls in doing 
research on auto-instructional technology, both with regard 
to his own work and to evaluating experiments by others. 
The results of a study which depends on a poorly written 
program should be interpreted accordingly. Programming 
is at least as much of an art, at the current time, as clinical 
psychology, for example. No matter how fine an experi- 
menter one is, he is not in a position to „evaluate clini- 
cal patients using a Rorschach or base a study on his 
interpretation of projective test protocols unless he has 
been thoroughly trained in clinical techniques. Similarly, 
despite how well the experiment has been designed, its 
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implications for a science of programming are limited if 
the program writers have not had sufficient experience. 
This is especially true when the sequence of frames is 
very small. A surprising number of studies have been 
carried out using a “program” only 50 or 60 frames long. 
The validity of the conclusions drawn from such experi- 
ments should be strongly questioned. 


PROGRAMMING STEPS 


Now to return to the problem of how programs are 
written. Because it is the area with which I am most 


familiar, I will describe the procedures we have been . 


using to program physics at the American Institute for 
Research. It should be noted, though, that the technique 
for preparing programs varies from individual to indi- 
vidual and, furthermore, that it is frequently modified 
because of the nature of the subject matter involved. 

Our first step is to list with care the specific objectives 
we will attempt to achieve with the program, expressed, 
in clear, behavioral terms. Most often, these objectives 
represent criterion behaviors, that is, the particular re- 
sponses we would like to see in the student’s repertoire 
when he completes the program. For academic material, 
criterion behaviors often can be expressed in terms of a 
comprehensive test covering the entire course. 

It is at this point that we make some initial decisions 
as to what kind of program will be constructed. For 
example, if the test (and the goal which it represents) 
requires only the ability to recognize the correct alterna- 
tive as on a multiple-choice test, the programmer will 
attempt to achieve only this goal. It is not necessary, then, 
for him to prepare frames requiring constructed responses 
similar to those in the preceding figures. In the same 
way, if the student is required to know how to use a 
particular formula but not remember it, the programmer 
will plan to teach him just that, and no more, It has 
been our expe-ience that, for academic subjects at least, 
the educational goal usually is to provide the student with 
the skills necessary to recognize a given principle when 
he comes across it, express the principle using his own 
words, and apply the principle to a new situation of 
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problem. Typical criterion behaviors which might be 
listed for a unit on heat, for example, might include such 
questions as: What is heat? How is it measured? What 
is the principle of linear expansion? Show how Boyle’s 
law might be applied, and so forth. 

Once the objectives have been identified, the next 
step is to carefully prepare a course outline covering the 
material that we wish to teach. The outline is frequently 
derived from a number of textbooks, reference sources, 
and from the comments of teachers familiar with the 
subject matter area. In addition, we have found that it 
has been highly desirable for us to obtain the assistance 
of a technical expert in the subject matter field. This 
expert is given the task of reviewing the materials pre- 
pared at each phase of the program’s development and 
is indispensable in preventing the dissemination of errors. 
We have also found that a well-qualified expert fre- 
quently can supply a large number of specific examples 
and interesting illustrations which can be incorporated 
into the program. 

_ The third step in our procedure is to begin the prepara- 
tion of draft frames. One of our programmers Is assigne 
to each major unit and he or she is responsible for 
drafting groups of frames which cover the subject matter 
contained in the outline in such a way as to achieve the 
behavioral objectives of the course. We find that the 
Process of drafting frames often points to inadequacies 
in the outline. Breaking down material into very small 
Steps frequently identifies not only omissions in the out- 
line but instances where it contains material which does 
not reflect the course objectives. ; 

The draft frames are then edited three times. The 
first editing is done by another programmer who attempts 
to simplify the program and discover errors which the 


original programmer has made. The second editing is 
done by the technical expert who reviews the frames to 
A third editing is done 


Insure their technical accuracy. ý 
by someone skilied in writing. Frames, just like text- 
books, can either be interesting or dull. Our third editor 
reviews frames, rewording those that are awkward or 
unclear and revising others so that the program is enjoy- 
able to work with as well as instructional. 
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At this point, the most important step in the prepara- 
tion of a program is begun. The frames are shown to a 
trial subject who proceeds through the program as a | 
student might eventually do in a classroom. Each answer — 
is carefully recorded and, on the basis of the data ob- 
tained, the program is revised as much as necessary by 
either adding new frames or rewording those already in 
the program. It is at this step, possibly, that the difference 
between auto-instructional techniques and what many | 
teachers do in the classroom is most pronounced. In 
programming, we assume that the burden of instruction 
rests with the program and not with the learner. Thus, 
when the student makes an error, it is assumed to be the 
fault of the program and not some inability on the part 


tepeated. In fact, 10 or 15 trial subjects are often used 
in the preparation of a single sequence of frames. The 
programmer is rarely satisfied that his materials are as 
good as they can be until trial students not only show 
that they have mastered the material in terms of how 
well they do on the criterion tests, but can proceed 
through the program without making an error. Follow- 
ing the last revision of the program, the frames are again 
reviewed by the technical expert and only then are they 
teproduced for field tryout. 

The most difficult step in this procedure is the actual 
writing of the frames. As I noted earlier, we have barely 
begun to understand what makes a good frame. We do 
have a number of leads, however, that can be derived 
from laboratory studies of animal learning. I have dis- 
covered that educators often have a particular aversion 
to discussing the problems of classroom education in 
terms of how animals learn in the laboratory, but watch- 
ing a rat, pigeon, or a chimpanzee learn can be a very 
instructive experience. The laboratory animal is not bur- 
dened with problems of low aptitude, inadequate readi- 
ness, lack of interest, or an unfortunate home life. In- 
stead, the experimenter accepts the animal as is and 
modifies his own behavior to the extent necessary to 
produce learning. 

The aspect of animal learning with particular impor 


of the trial subject. 
When the frames are revised, the trial procedure is 
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tance for preparation of an auto-instructional program 
is the methods used for initially generating the correct 
response. Not only are these methods important with 
Tespect to auto-instruction, but perhaps one of the prime 
objectives of research in education today is the discovery 
of techniques for getting appropriate responses to occur 
initially. Probably this is our most ineffective skill in 
education and training. In teaching swimming, for in- 
stance, it is often quite difficult to get the correct response 
to occur the first time. The same is perhaps equally true 
of physics or spelling. The fewer similar or equivalent 
Tesponses the student has, the less likely the responses are 
to occur at the appropriate time in the training schedule 
for practicing and reinforcement. r 

In the laboratory, we often wait until the animal 
makes a response before reinforcing him. We have 
found, though, that it is not necessary for the animal 
to make a complete response, but only make responses 
which are preparatory or similar to the desired response. 
f we reward these preparatory responses, a procedure 
called successive approximations, we speed training since 
the animal is taught behaviors which make the criterion 
behavior more likely. P d 

Another procedure is to control the environment in 
such a way as to prevent erroneous responses from 
Occurring. For instance, we can reduce the size of the 
apparatus until almost the only response that the animal 
can make is the desired one. The third possibility is to 
build on the animal’s previous learning. We can teach 
a rat to run through a complicated maze quite readily if 
We first train the animal to follow lights and then place the 
lights appropriately in the proper alleys of the maze. By 
and large, these procedures are not systematically em- 
ployed in education. For example, while we try to limit 
the environment in the laboratory, teachers are often 
encouraged to decorate their classrooms and provide other 
stimulation which, in fact, produce unwanted responses 
rather than eliminating them. I have always been struck 
by the amount of time a teacher spends in carefully dec- 
orating her room and the subsequent difficulty she has 
in maintaining the attention of students who are at- 


tracted to the displays she has put up. 
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Despite the rather obvious implications these labora- 
tory studies have for improving instruction, a surprising 
number of programs have been constructed on the philos- 
ophy that the student is bound to make errors while ~ 
learning. These programs typically incorporate a feature 
which is called branching, that is, alternate sequences 
to be used by the student after he has made an error. 
The same philosophy leads to the construction of devices 
which are capable of preventing cheating. One wonders, 
though, whether branching or cheating must be a problem 
in education. After all, since we want the student to 
practice being right, why not help him as much as we 
can? In preparing auto-instructional materials, then, the 
programmer generally makes every effort to carefully 
guide the student through the required learning. He de- 
signs the program so that it plays an active role in 
insuring that learning will take place and that the proper 
responses will be those that are practiced. 

This particular approach to learning, which depends 
largely on placing the pupil in a carefully controlled 
environment, is not necessarily alien to education. Credit 
for the suggestion that instruction might be automated 
by properly arranging the circumstances in the student's 
environment is often given to Socrates or Pressey or 
Skinner. It is my own feeling, though, that many of our 
current notions on auto-instruction were accurately antici- 
pated by John Dewey. To him, learning came about by 
doing; not random doing, however, but doing under 
such careful control that the student would necessarily 
meet with success when he was exposed to the conse- 
quences of his actions. Controlled or guided practice 
which leads to reward is the central theme in both 
Dewey's approach and that used in auto-instruction. 


PROGRAMMING RULES 


Now, let us look at some of the rules we have for 
building frames. While these rules generally represent 
the theory of auto-instructional programming rather than 
experimental evidence, they illustrate the auto-instruc- 
tional method and thus might be a help at this time. The 
first is, that the student learns from making a response 
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and not from hearing or seeing it. Only by practicing 

a response will a student learn and retain it. 
p In programming, the question often arises as to whether 
it is important that the student make his response overtly 
or whether it is sufficient for him to make a covert 
response. With respect to simple verbal material, there 
appears to be no difference produced by these methods. 
On the other hand, if the goal of the program is to 
develop some manual or perceptual skill, active respond- 
ing seems to be very important. The same is true when 
the student is apt to over estimate the ease with which 
he learns. Only when the student writes out a new and 
unfamiliar word does he receive the necessary practice in 
spelling it. Pronouncing the word to himself will not 
likely yield as satisfactory a result. 
Another question that often arises is whether to re- 
quire multiple-choice or constructed responses. Since they 
are capable of doing so, programs can and should encour- 
age the student to express his ideas in his own words. We 
then are sure that the student's practice is of the kind 
We usually want to generate; by means of constructed 
Tesponses we require the student to use and explain what 
e knows so that he practices presenting his ideas and 
his knowledge instead of simply recognizing a correct 
answer from among the alternatives given. There is still 
one more advantage to have overt, recorded responses. 
Ns is to provide the programmer with the data neces- 
sary for further revision of his program. The first tule of 
Programming: require active responding. À 
4- The next concern of the programmer 1s how to insure 
that the student will make a correct response, especially 
the first time that particular response is required. Aside 
from the laboratory methods described earlier, the pro- 
grammer can get assistance in this matter by studying 
the techniques used in advertising. In most instances, 
the advertiser is primarily concerned with producing the 
Initial response. Some of the more effective advertising 
we have is based on the contiguity learning principles of 
Guthrie and Estes. Their theories suggest that a given 
Tesponse can be readily produced providing we mass 
sufficient cues and present them all at the same time. 
nen driving down the highway, a motorist is apt to 
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come to an intersection where his stopping is made 
rather certain due to the presence of a stop sign, 4 
flashing red light, and appropriate yellow lettering on 
the pavement. i ; 

In programming, each frame must contain sufficient 
cues to produce the desired response. Sometimes we 
can get the proper response to occur by the use of 
wording; at other times, we may use thyming, synonyms, 
or particular word patterns we can assume are already 
in the repertoire of the students involved. , 

\ Appropriate cueing is one of the essential charac- 
teristics of a good frame. In fact, this is the essential 
diference between a test question and an auto-instruc- 
tional frame. In giving a test, we design the items in it 
to at least partially lead the student astray. We don’t 
want every student to get every answer correct. In a well- 
constructed program, however, every effort is made to 
insure that every student gets every response correct. 
The second rule: proper cueing. 
_ The fact that a student can make a correct response 
is rarely of interest to us unless the response is made in 
the appropriate context. Little Johnny may be doing an 
excellent job of pronouncing “daddy,” but after the first 
few times we hear it, we fail to think much of this 
response unless it occurs in the presence of the proper 
man. As learning proceeds during the course of using an 
auto-instructional program, the student must be gradu- 
ally weaned from the liberal cues we have supplied, so 
that he can make the correct response without our help 
in the presence of the relevant context. Third rule for 
writing a good frame: appropriate context. 

Together, these first three rules can be combined to 
describe a good frame. Basically, a good frame consists of 
only four parts. First is the response, selected beforehand. 
This is the part of the frame that must be left blank- 
Too frequently, a beginning programmer finds that even 
though he has selected a satisfactory response, he con- 
structs the frame in such a way that some irrelevant 
word rather than the desired response is practiced. The 
second portion of a good frame contains whatever cues 
are necessary to reliably produce the desired response 
Neither overcueing nor undercueing is desirable. The 


| 
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` third characteristic of a good frame is that it contains 
aene televant context, that is, the kind of stimuli we 
ope will be capable of evoking the desired response at 
some time in the future. Early in the program, context 
will probably be at a minimum and cueing will be quite 
+ prevalent. By the end of the program, however, cueing 
should have vanished so that the appropriate responses are 
made solely in the presence of the appropriate context. 
The fourth ingredient of a frame is optional. We call it 
enrichment. We have found that, in many instances 
Interest in a program is heightened by inserting facts such 
as names or dates which are relevant to the material being 
covered but not part of the course objectives. Aside from 
these four ingredients, everything else in a frame is 
Superfluous. i 
The next characteristic of a good program is that it 
proceeds in very small steps. This is a rather difficult 
point to get across to a new programmer. Several studies 
have indicated, however, that the more steps, the better. 
In fact, the data seemed to indicate that a student 
proceeds faster through a program purposely made longer 
by the division of frames into smaller units than he does 
through a very condensed program. The number © 
frames in a program is not necessarily a good index of 
its efficiency, but as is the case in many other areas, 
one seems to get what one pays for, that is, the more 


frames the better. Rule four: small steps. 
veloped, the role of careful se- 


id 
tá As the program is dev j 3 
quencing becomes evident. Most subject matter consists 
ncepts are 


of material of unequal difficulty. Complex co 
built on simple concepts. Long involved formulae often 
can be shown to represent several shorter formulae com- 
bined together. In preparing a program, the writer can 
greatly enhance its efficiency by thoroughly determining 
beforehand what the sequence of topics will be. The 
fifth rule of programming: careful sequencing. 

Another problem having to do with the,program as a 


\-whole is the problem of how often and how frequently 
d. As was the case with the 


a response should be practice 

number of steps, the rule here seems to be that the greater _ 
the number of repetitions, the better. The repeated review 
of a given"response is almost universally more irritating 
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to the programmer than it is to the learner. In the first 
place, the program writer already knows the response. To 
him, frequent repetitions are terribly tedious. From the 
student’s point of view, however, repetitions can be ter- 
ribly important. It is one of the basic principles of 
learning that retention of a response depends quite heavily 
on the amount of overlearning that has taken place. Thus, 
when a student complains of having made a certain 
response too often, we should be careful not to delete so 
much practice with that response as to affect its retention. 

One guide to assure that repetition is frequent enough 
is that the programmer, in reviewing his own frames, 
should be sensitive to his echoic behavior. When a par- 
ticular response stops ringing in his ears, it is probably 
time to review that response in the program. It is prob- 
ably helpful, in addition, to introduce as much variation 
as possible in the cueing and context associated with a 
particular response each time it is repeated. This kind 
of variability can be expected to reduce monotony and 
also to induce a desirable amount of generalization with 
tespect to that response. The sixth rule for programming: 
frequent repetition, 

The next rule concerns the technical accuracy of the 
material contained within the program. The preparation 
of an auto-instructional program is a great amount of 
work. Too often, a writer engages in the necessary effort 
but fails to realize that the very excellence of his program 
in a psychological sense insures that his students will 
learn well, but learn the wrong facts and concepts. In- 
suring technical accuracy, of course, also involves the 
inclusion and exclusion of material. It is often the case 
that a programmer attempts to include too much into 
his program to permit the proper coverage of any one 
topic. In general, it is probably more advisable to do a 
thorough job on somewhat less material than an incom- 
plete job on a larger amount. The seventh rule: knowl- 
edge of the subject matter. 

ow we come to a few don'ts, Probably the most 
common mistake made by a beginning programmer is 
that he prepares the frames as if he were a lecturer 
presenting information. 


e lecturer can be contrasted with the tutor or auto- 
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instructional program in that he guides the student in 
what to learn but then delegates to the student the 
ra apa for structuring the material in small steps, 
or sequencing these steps, for scheduling sufficient 
— and for meeting with the success that will serve 
to reward desirable responses. Imitating a lecturer, the 
inexperienced programmer writes a paragraph for the 
student to read and follows it with a question which the 
aap is expected to answer. Frame A shown in Fig. 
is an example of this type. 


Temperature and heat have different meanings. 

Heat, the amount of thermal energy stored in an 
object, depends on the mass of the object and what 
Frame A it is made of as well as its temperature. Heat is 
measured in calories, but since heat and temperatyre 
are not the same, we do not me 


in calories. 


SUE wee eeeneeeee 


In general, liquids, solids, and gases will expand 


Frame B when heated. Kerosene is a ssseseresererseeeel 
it will expand when heated. 


mm 
Most metals contract when they solidify, but there 
Frame C are exceptions to this rule. For example, bismuth 
and antimony do ....sss.strtt" maese.. contract 
when they go from a liquid to a solid state. 
— 


Copper, iron, wax, and glass all contract when they 


Frame D | solidify. Most substances, in fact, « - 
when they go from a liquid to a solid state. 


An astronomer, Anders Celsius, invented the centi- 
grade scale of temperature. A thermometer which 
Frame E reads 100 degrees in boiling water is measuring 
the temperature of the water according tO the 
. scale of temperature. 


rature is more prop- 


The centigrade scale of tempe! 
scale, after 


Frame F erly called the ..scssessereseeee® 
the name of its inventor. 


Fig. 2. Six frames illustrating weaknesses and strategems in 


programming. 
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Programmers lecture in frames when they are not sure 
what they want to teach, when they have not clearly 
identified the specific objectives of the frame. Analyzing 
this frame in terms of cue, context, and response, we 
see that it is quite possible that a student could answer 
the frame solely on the basis of the last sentence. The 
other material in the frame, representing facts which 
the programmer might want the student to know, neither 
helps produce the response called for nor does it provide 
the kind of context in which we want the response of 
temperature to necessarily occur. In programming, we 
not only provide facts to the learner, but we help him to 
learn. If the student doesn’t understand, it is our fault, 
not his. This is the eighth rule of programming: don’t 
lecture, teach. 

Frame B illustrates another very common error made 
by beginning programmers. Here’ the programmer had 
the beginnings of a quite satisfactory frame, but after 
he had written out a statement, he then selected a rather 
trivial word to be filled in by the student. The frame 
could be improved tremendously had he left the word 
“liquid” where the blank is now and removed the word 
“expand.” It is the concept of expansion, after all, which 
the programmer was apparently attempting to teach. 
Advanced programmers sometimes make almost an op- 
posite error. They decide that because the word or re- 
sponse itself seems trivial, it is not important enough 
to be left blank. 

Frame C illustrates a frame that calls for such a 
response. Here, the word “not” is important. The fact 
that bismuth and antimony do not contract is the point 
we are trying to get across. The ninth rule of program- 
ming: evoke a relevant response. 

Frame D illustrates another one of the rules. It illus- 
trates how we can frequently depend on rather subtle cues 
to insure that the appropriate response will occur. It was 
not necessary, in this instance, to tell the student pre- 
viously that most substances contract when they solidify. 
Instead, we have let the student make his own gen- 
eralization and discover his own principle. The tenth 
tule: don’t provide more cues than necessary. 

Now look at the next frame (Frame E) in Fig. 2. 


pm 


pa 
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This frame, while not especially well written, does get 
its point across but it also illustrates another important 
error. The programmer may have assumed that his 
students would recognize the relationship between 100 
degrees, boiling water, and the centigrade scale of tem- 
perature. I suspect that if we tried this frame on a 
large number of students, we would find that several of 
them would make the error of inserting the word Fahren- 
heit (if they could spell it) into the blank. This is our 
eleventh rule: don’t assume too much knowledge. 

In the last frame in Fig. 2 (Frame F), we can see that 
the author failed to recognize that we learn from respond- 
ing and not from reading. It is very unlikely that the 
response Celsius would be evoked here in the average 
student unless he had previously learned the astronomer’s 
name. Recalling the preceding frame, we can see one 
possible source for the programmer's error. He attempted 
to present too many facts within a single frame. The 
twelfth rule is: don’t present two new facts in one frame. 

These twelve rules sum up much of what we know 
about the art of programming at this time. As noted 
carlier, they tend to be incomplete, and we have not yet 
achieved our objective of being able to provide firm di- 
rections to beginning programmers. The lack of definitive 
tules, however, does encourage many programmers to 
experiment and thereby produce new rules and new 
concepts of programming. Perhaps one of the greatest 
errors that a programmer can make is attempting to copy 
the kinds of programs he has seen before. His objectives 
will frequently be too minimal, his program uninteresting, 
and he will fail to produce the kind of learning that he 
feels should occur in the classroom. There is no reason, 
for instance, why the blank in a program shouldn't 
Tepresent a sentence or two. As another example, there 
1s no reason why the programmer can’t employ pictures 
—or for that matter, even have the students draw their 
oin. We have used both of these kinds of responses 1m 
our physics program. 

The lage tne frames I would like to illustrate both 
Tepresent instances where the student is required, in 
terms of the way the frame is structured, to create a 
response which he may never have seen before. 
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The first frame in Fig. 3 (Frame G) is an example of 
this type. Here the word “also” cues the response. The 
student may never have seen the word “Kelvin” before, 
but yet he is able to respond that Kelvin is a scale of 
temperature. This frame also illustrates the use of the 
instructional approach which I have been describing. In 
programming, the primary objective of each frame should 
be the active participation, of the student in terms of 
carefully guided or controlled practice. An auto-instruc- 
tional program should not attempt to explain a prin- 
ciple to a student. Instead it should carefully lead him, 
one step at a time, toward the desired level of proficiency. 
Here, in this frame, we have done more than could be 
accomplished by presenting the fact that “Kelvin is a 
scale of temperature” to a student; we have forced him 
to actually practice the correct response, in the proper 
context, with a minimum of cues. 


Centigrade and Fahrenheit are both scales of tem- 


Frame 
ramen perature. Kelvin is alsoa* ......seseeeeeeeees 


Knowing that most metals expand when heated, 
and remembering that the period of a pendulum 

Frame H depends on the length of the rod, we would expect 
that a pendulum clock Would ......seceseeeeeee 
time on a cold day. 


* More than one word belongs in this space. 


Fig. 3. Two frames that lead to discovery by the student. 


The next frame (Frame H) illustrates further the 
non-tote quality of learning possible to achieve with an 
auto-instructional program. In a frame of this sort, 
which ordinarily would have been preceded by steps 
which taught the two facts appearing in the first part of 
the frame, the student is encouraged to discover some 
new fact about the world around him. In neither this nor 
the preceding frame do we depend on explanations. 
Instead, we have tried, subtly, to guide the student and 


help him experience the satisfaction of being right, all 
on his own. . 
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r One of the greatest contributions which programming 
is likely to make to education is the clear realization and 
understanding of the learning and instructional processes. 
I suspect that concepts such as the careful control of 
responding to insure its initial accuracy may improve 
classroom teaching whether or not auto-instructional 
programs have been devised. 

What I am saying is that the means for placing 
materials on paper is not as important as our under- 
standing of the instructional process. To illustrate, not 
too long ago I read somewhere of a technique used at a 
school where every effort was being made to teach the 
children cooperation as opposed to competition. While 
this only represents one small part of the training that 
would likely be necessary, I feel it was an ingenious 
solution to provide the children in the pre-school play- 
ground with an assortment of toys of the kind that 
required the cooperative activity of two or more children. 
That is, teeter-toters instead of swings, and wagons SO 
large that the strength of two children was required to 
move one. This example of programmed learning of a 
social skill or habit illustrates the point that while it 
may not always be feasible, practical, or economical to 
provide instruction independent of a 1 
there seems no reason to believe that if a skill, fact, or 
concept can be taught, it would not be possible to achieve 
it using auto-instructional methods. 
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EDITOR’S NOTE 


Predictably, the first studies of pro- 
grammed instruction tended to be evalu- 
ative; i.e., the purpose of the studies was to 
compare programmed materials and con- 
ventional instruction on their efficiency as 
teaching devices. Quite predictably, for the 
majority of such studies, “no significant dif- 
ferences” was the conclusion. Once that it 
had been determined that programmed ma- 
terials could teach, attention was directe 
toward research which would isolate some 
of the parameters of learning with materia 
of this type. Only a relatively small amount 
of data is available at the time of writing, 
but it is sufficient to point the directions 
which research will take in the immediate 
future. Some of the areas for which more 
information is needed are: 


u. What are the (1) characteristics of the 
“ideal” frame or step? This can be subdivided 
into (a) What are the characteristics of the 
“ideal” stimulus elements? (b) What are the 
characteristics of the “ideal” response element? 
(c) What type of confirmation technique is 
best? and (d) What type of activity will 
mediate best between the stimulus and re- 
sponse elements? 
_ What type of programming technique results 
in the greatest gain, with time to learn held 
constant? a 
How well is knowledge acquired from pro- 
grammed materials retained? d 
Is it necessary to develop programs which 
take individual differences in ability and mo- 
tivation into consideration? 
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Part Ill 


What type and quality of reinforcements 
should be used with programmed instruction? 

What are the best criterion measures for 
evaluating learning from programmed mate- 
rials? 

What characteristics must a program have 
to insure satisfactory transfer of training? 

What are the criteria needed to evaluate the 
quality of a program? 

What learning models, if any, provide a 
satisfactory frame of reference for further de- 
velopments in programmed instruction? 


The articles which have been selected for 
Part III deal, in part, with the first five of 
the research problems listed above. The first 
five articles are concerned chiefly with 
knowledge of results as a confirming tech- 
nique and the effects of the mode of re- 
sponse on learning. Two articles are con- 
cerned with the problem of individual dif- 
ferences and one with stimulus characteris- 
tics of the frame or step. 
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Development and Appraisal of Devices 
Providing Immediate Automatic Scoring 
of Objective Tests and Concomitant 
Self-Instruction* 


S. L. Pressey 
Ohio State University 


A. “SELF-SCORING” TESTS: THEIR UNIQUE 
CONVENIENCE, AND USEFULNESS IN INSTRUCTION 


The grading of conventional essay-tyPe examinations 
has long been recognized as burdensome and slow. By 
contrast, the scoring of objective tests 23 
and prompt. But actually this latter task is 
negligible, or as immediate as might be di 
y multiple choice Ot true-false. 


tests are most commonl 
The person taking the test checks the answers he con- 


Siders right, on the test blank or on 
sheet. The materials are then collected, a 
time the errors are checked and counted. 
or simple testing devices could be developed such that, 


a 


* a 
The work described in_this papet was under subvention 
from the Special Devices ae: the Office of Naval Re- 
search, for development of devices for tho facilitation of 
earning, working in cooperation with the Ohio State Univer- 
Ty Research Foundation, under the dir vo 
w article is reprinted from The Journal of Psychology, 1950, 

, 417-447, and is used here with the permission © the 


author and the editor. 
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as a student answered each question, that answer was 
immediately and automatically scored and recorded as 
right or wrong, then clearly much trouble would be saved. 
Moreover, results would then be available as soon as the 
test was finished. If a future emergency should necessi- 
tate the rapid training of large numbers of men, such 
convenience and immediate availability of scores could 
appreciably facilitate the training. . 

Additional advantages of self-scoring devices are more 
important, however. A test in French or mathematics or 
navigation is of little value if it does not in some way 
further the student’s learning of that subject. If he is 
weak on certain points, the test should locate them and 
aid in the remedying of these weaknesses. And this 
should be done promptly; an instructor who never an- 
swers a student’s question until 48 hours after it is asked 
would be considered exasperatingly inefficient. The usual 
testing methods are grossly at fault in all these respects. 
Usually, the student does not find out how he did on a 
test until a day or more after it was taken. Even then, he 
may be given only his total score or grade, without any 
indication as to questions missed. If the scored test 
blanks are returned to him, he may be little or no better 
off. Even if his errors have been marked on the test, he is 
still left in ignorance as to which other answer on a mul- 
tiple choice question is right. If separate answer slips are 
used and these returned checked but without the test 
blank, he knows for instance that he missed Questions 12 
and 19 but does not know what Questions 12 and 19 
were.! In contrast, self-scoring devices inform the student 
immediately, when he indicates his answer to an objec- 
tive question, whether that answer is correct; if it is 
wrong, he is at once automatically guided to the discovery 
of the correct answer. This quick incisiveness should sub- 
stantially facilitate instruction. In an emergency, requit- 


* The IBM electric test scorer is little better in the above 
respects than hand scoring. Results are still not available until 
a day or so after the test is taken. Some inspection of blanks 
is often necessary to be sure that students have marked each 
answer space adequately and not marked more than one for 
each question. Each blank must then be fed through the 
arhe and its total score written on it. And the scored 

lank does not show in any way which questions were missed. 
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ther: regarding the nature of the learning process that 
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ade pore A Theories of learning recognize the 
and obvious importance of the learner's 


owing whether each response he makes is correct or 
effec pane has shown that such knowledge is most 
dicat if obtained without delay- There are data to 

ate that answers found to be wrong, but not cor 
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inn tend to be repeated. Other things being equal, 
esponse which has been made most often, and most 

in. The devices 
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TREN is most likely to be made again- 

ggested above provide that the learner always is in- 
nswer to every 


f 

Pere about the correctness of every a 

gestion on a test. That information is given immedi- 
y. If a mistake is made, the learner is at once guide 
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ion; and if the test is 


answer before leaving the questio: 
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ch learner's weaknesses 


met oot quickly and comp 
aia o, so such devices spot ea a 
atel ma their correction more adequately and immedi- 
he han could any human instructor. 
ae ie instructional device, the self-sconng test has cet- 
ih urther advantages. An objective test can covet many 
Hon Se in a given time than a written examina- 
similarh quiz. The self-instructional objective test can 
Rive] y cover more ground than would be possible Or. 
eale a given time, it can 1M effect give individualiz 
aes ions to every student in 4 class ( 
them} ceptions of each student and 
ane ie more questions than a tutor 
full ent. A well made objective test has a more care- 
y considered and exactly phrased right answer than 
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an instructor is likely to give in class discussion; and the 
wrong answers are carefully chosen as the most common 
misconceptions or mistakes. The self-instructional pro- 
cedure systematically presents these for consideration and 
then clearly checks them off as wrong. In short, these 
devices should make it possible to cover more ground in 
a given time than a teacher could and do it more ade- 
quately. 

The purpose of the present paper is, briefly to (a) de- 
scribe means by which tests may be made self-scoring and 
self-instructional, (b) investigate the way in which such 
devices work, and (c) determine their value in educa- 
tional or training programs. 


B. MATERIALS AND METHODS 


1. The “Self-Scoring” Device 


The total project has involved development or im- 
provement of several different devices.2 However, the 
data of the present paper were all obtained with the 
“punchboard.” This is a very simple unit the size of a 
three by five card, about one-half inch thick. A center of 
quarter-inch ply-board has riveted, on either side, two 
thin sheets of pressboard. Between the outside sheet and 
the second sheet is sufficient space for a slip of paper to 
be inserted; between the second sheet of the pressboard 
and the ply-board center is a removable key sheet of 
pressboard. On each face of the punchboard are two 
columns each of four rows of one-eighth inch holes, the 


* The basic idea involved in all of them goes back to work 
by the writer in 1915 and was embodied in a mechanism for 
use with objective tests in 1922 (Pressey, 1926). It is believed 
that this work had some influence in stimulating efforts © 
other psychologists along these lines. The present project aime 
originally at the development and appraisal of devices im- 
Proving on these mechanisms. However, pending availability 
of these new pieces of apparatus, it seemed desirable intensively 
to investigate the value of self-scoring procedures. The punch- 
board was therérore devised and used for this purpose. No claim 
is made to originality regarding the “punchboard” method, the 
general idea of which goes back at least 20 years. Recently Dr. 
Maurice Troyer (Angell & Troyer, 1948) has been experi- 
menting with a similar scheme. Other devices for immediate 
self-scoring will be touched upon later, 


| 
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tows of four holes being numbered from one to 30. How- 
ever, the key sheet has holes only for the right answer on 


Test questions are usually presented on a mimeo- 
graphed sheet, but may be given orally, put on the black- 
board, projected on a screen, or otherwise presented as 
convenient. The testee takes a test by simply pune 
with a pencil point through the p 
which corresponds to what he t 
For instance, if for Question 1 of the test he t ee 
second answer is right, he will punch his pencil throug 
the paper appearing in the second hole of the pee 
If he is right, his pencil goes through the paper and down 
into the hole in the key sheet. Bu a 
pencil barely breaks the paper and then comes up ag: 
the key sheet. He thus knows that he is wrong B 
another hole in Row 1, thus proceeding until the pen 


goes deep. 

After ia has gone througit a test, he can ganes bie 
and see on what questions he made a mistake (where 
punched more than one hol 
be done either before Hi her before or 
rom the punchboard; and either r 
easy to te up the number of small holes to obtain m 
error score. A space at the top of the front plate ? até 
punchboard gives room on the paper for the rs e i 
write his name and other information as well as the err 


“Th is no possibilit 
The device is very simple to use- There is no pP : EY 
of changing an answer (a hole cannot be auie a 
The punched papers, once withdrawn, can A e ar 
are a convenient record for ysis or ot ia si £ 
The keys are so planned that each can be tume over 
turned end for end, and thus give four 
Two or three different key ca 
variations in pattern so that 
unlikely. In setting up 2 test, the p 


the pattern of right answers to CO 
the ready-made keys, which have been carefully planned 


to distribute the right answers’ oveT the four pene 
However, it is relatively easy to punch a new key 
is desired. For true-false OT other tw 
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columns can be used on one side. The back side of the 
punchboard, which is made exactly like the front side but 
numbered “2,” either permits a second trial on the test 
to find improvement and give more practice, or provides 
a total of 60 multiple-choice items. 


2. The “Self-Instructional” Tests 


The tests used with the punchboard in the experimen- 
tation to be reported almost all consisted of four-choice* 
objective questions, of which samples will be given 
shortly. Tests which consisted of a series of statements 
to be appraised as true (by punching the first hole) or 
false (by punching the second) were compact and rela- 
tively easy to make, but seemed somewhat less effective 
for instruction. Tests of this same type but with a third 
category of “uncertain” (shown by punching the third 
hole in a row) were tried but were hard to make, since 
defensible differences of opinion as to best answer them 
often appeared. A variety of other judgment, applica- 
tional, and special types of items were tried extensively. 
Some were believed to have distinctive values for self- 
instructional testing. However, they presented difficulties 
of one sort or another, and will not be described here. 

Much consideration was given to the problem of de- 
sirable length of tests. Early plans were for 100 questions- 
However, for instructional purposes, frequent short tests 
were found better than more occasional long ones. In- 
structional tests were found most effective if followed by 


° Early experimentation was with five-choice questions. How- 
ever, it appeared that the great majority of items simply di 
not have four wrong answers which were all statements 0 
errors that students were likely to make and needed to be 
warned against, or were otherwise justifiable in an instructiona 
test to be used as these devices required. Where the most im- 
Portant function served by a test is instructional, each alter- 
native answer should make a real contribution to instruction. 
Each wrong answer should be one against which a warning 
is needed, or which elucidates the question in some way- 
alternative answer should confuse the student or introduce ways 
of construing the question which are not educationally profit- 
able to consider. Poor alternatives waste time both in taking 


the test and in discussion f i 7 
arne! 
rather than help him, PAE and SETS BE 
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discussion the same hour. Even with so short a task as 
a 30-item true-false test, several runs through it with the 
punchboard were often found to be needed, if a perfect 
score was to be reached. ‘All these and other considera- 
tions put together led to the conclusion that a self- 
instructional test of only abort 30 or 40 questions was 
better than any longer unit. Moreover, such short tests 
had obvious practical advantages. With a little careful 
planning, a test of 30 four-choice questions could be put 
on one side of a single long mimeograph sheet, or 30 
true-false statements on one side of a short sheet, and 
almost any test of that many items on two sheets or two 
sides of one. In addition, the punchboard could be kept 
small.- For use on the arm of the usual classroom chair, 
such compact convenience was a distinct asset. | 

Initial experimentation was with a great variety of 
subject-matter ranging from nonsense syllables, yocabu- 
aries in a foreign language, and glossaries of naval nae! 
to technical articles and chapters from books used in the 
Naval R.O.T.C. training program at the University, 
and series of tests covering the required reading 1m two 
courses in psychology. It was finally decided to concen- 
trate on material of three types. Russian vocabulary was 
used as rôte matter more interesting and valuable than 
nonsense syllables, equally unfamiliar to almost every- 
one, and indicative of possible values of self-instructiona 
tests in learning a foreign vocabulary. Hard English vo- 
cabulary items were used to represent tasks such as the 
learning of technical terms where the material was highly 


Meaningful but (in contrast to the subject-matter ie 
Course) with canization or structure. ‘1 
soapy Bo rses served as an 


subject matter of two psychology cou 
example of material hs was highly. organized, an 

part of a systematic program of instruction. The ollie 
ing excerpts from certain of these tests will illustrate the 
simple directions used and some typical items. The psy- 
chology test excerpt includes two features found dis- 
tinctly useful for review study after a test is taken: head- 
ings in the test to give it structure, and page references 
So that if a student misses 4 question and wishes to look 
it up, he can turn at once to the place in the reading 


where that topic is treated. 
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Russian Vocabulary Study Test 


Directions: In each line find the English word meaning 
the same as the Russian word, and punch your pencil into 
the hole corresponding to that English word. If your pencil 
goes deep, you are right, and should go on to the next line. 
But if your pencil barely breaks the paper, you are wrong, 
and must try again—until you do find the right answer. 

l. ZapisKa: (a) plate; (b) expensive; (c) note; (d) 
bridge. 

27. Porom: (a) depth; (b) then; (c) paper; (d) word. 


English Vocabulary Study Test 


Directions: Following each word, find the phrase you think 
nearest in meaning to it, and punch your pencil into the 
hole on the punchboard corresponding to that answer. If 
your pencil goes deep, you have the right meaning. If it does 
not, try the phrase you think next most likely to be right. 
Continue until you find the right answer. Then study that 
word so that you will know its meaning, if you are tested on 
it again. 

l. INTEGUMENT; (a) a completed whole; (b) a covering 
as the human skin; (c) moral soundness; (d) a factor, as in 
mathematics. 

19. INimrcat: (a) favorable; (b) unique; (c) influential; 
(d) hurtful. 


Psychology Study Test Abilities 


Directions: Punch the hole indicating what you think the 
best answer to each question; if wrong, try again. 

Special Ability: 12. A person superior in art usually has 
(a) poor social adjustment; (b) a good general ability and 
background of special opportunities and motivation; (c) a 
highly special innate capacity; (d) lack of ability in other 
areas. 68-1. 

Appraisal of Ability: 24. The general ability of a person 
from a non-English speaking home may best be estimated by 
a (a) personal interview; (b) Binet test; (c) performance 
test; (d) group intelligence test. 77-1. 


For some of these tests, several forms were available, 
using different keys. For the two courses, sufficient variety 
im material was sought to give interest and also prevent a 
routine approach. Thus for one topic there were two 
four-choice tests, a true-false-uncertain judgment test, 
and an application test consisting of a series of pata 
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graphs each presenting a practical situation or episode 
followed by questions as to ways of best dealing with 


such a situation. There were also various check tests to 


determine the effects of the practice tests, which will be 
described later as used. 


3. Groups Experimented With 


At the Ohio State University the first and second 
courses in elementary general psychology and the first 
course in educational psychology are taught in sections 
of about 35 students, mostly freshmen and sophomores. 
n a given quarter there may be a total of as many as 
sections. Most of these are taught by part-time instiuc- 
tors in their third or fourth year of work for the doc- 
torate. Each course is under the supervision of a senior 
Staff member; there is a regular calendar, certain objective 
examinations are taken by all sections, and there are fre- 


quent staff meetings. These numerous sections, thus CO 
ordinated and systematically appraised, offered unusual 


pportunity for the extensive grou experimentation de: 
Sable in ‘his project. Further, it was possible in the 
Course in educational psychology to make sweeping 
changes in methods and hours of meeting when mf g 
or experimental purposes. Most of the data are trom 


pas of these sections. However, studens i 
3 -tC- orato: 
courses, from a how-to-study labori ry saline 


we on probation to certain graduate acre 
ae total investigation at one time oH = a 
s ar schoo 4 i 
three quarters of the regul his investiga- 


pber of cases, sections, and facilities for t 
On seemed exceptional. 


C. RESULTS 

i m attempting to appraise the value of devices involv. 
ing immediate automatic scoring and self-instruction B 
Natural first step was to determine whether, 10, a aA 
ing a test by such means would at once bring S 
learning, A natural next step was to investigate whe 


such dev; : Jar instructional oF 
e ji regular 
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taining programs. A desirable third step, W 
whether Dros tests might be exceptionally useful 
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with special groups or instructional plans. These issues 
wil be taken up in crder.* 


1. Immediate Outcomes of Use of the Punchboard 


As mentioned earlier, Russian vocabulary, hard English 
vocabulary, and subject matter in Psychology (as illustra- 
tive of respectively, rote matter, meaningful but un- 
organized material, and material organized and meaning- 
ful) were chosen to show how the punchboard worked 
with matter of various types. Certain immediate out- 
comes of such use are now to be considered. 

a. Effect of the punchboard on range of test scores. 
Ordinarily a person taking a multiple choice test checks, 
for each question, the answer he thinks right; and his 
total score is the number of questions answered correctly, 
or answered incorrectly. In interesting contrast, a self- 
scoring procedure such as the punchboard can measure 
degrees of ignorance for each question. For example, on 
a four-choice question a student could make no errors, 
or one, two, or three. On 30 questions, the total possible 
errors could range from zero to 90 as compared with 
zero to 30 for the same test taken in the usual fashion. A 
procedure which increases the range of scores three times 
should discriminate more widely between those students 
who had the subject-matter well learned and those who 
still were grossly ignorant or had many misconceptions 
to correct. 

A very simple experiment suggested that this was so. 
One college class was given a hard English vocabulary 
test in the usual way; each student checked the answer 
to each question which he thought right, and the score 
was the number of questions wrong. Lowest number of 
errors was 18 and highest 28, or a range of 10. Then 
another similar class was given the same test using the 
punchboard. Lowest number of errors was 19 and highest 
53, or a range of 34! It would surely seem that a range 
of 34 would discriminate good from poor students more 
agreeably than a range of 10, or less than a third as 
much. Similarly the range on a 30-question test in psy 


* Acknowledgments for work on various phases of the total 
study should be made to Dr. Viola Cassidy, Dr. Leslie Brigg’ 
Mr. Daryl Severin, and other assistants. 
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ue to th he English vocabulary might be explained as 
asic Ho EN ‘nitial difficulty of these terms plus their 
Punchbo: iliarity, making them easily learned, once the 
ard gave reminders as to their meaning.® 
or scores on the Russian test 


robably lower than they 
the results in hard English 


Vocabul 
not th ary and Psychology. But important conclusions appear 
Indicat imilar results 
cd that the punchboard brought improvement in two 
it the first time 


this choice as tight. 
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TABLE 1 


Learning from taking a test with the punchboard, 
as shown by decrease in errors on second testing. 


Subject matter of 


tests: Russian English Psychology 


Method of giving As Punch-| As Punch-| As Punch- 
tests: Test board | Test board | Test board 


Median? number of 
errors 
On first test 
On second test 


L] 

= | œx 
en 

Ga oN 
w 
o 
a 
m 
e 
N 


Decrease in errors 


a The median is the point below and above which there are equal 
numbers of cases. Classes averaged 31 students in size. 


c. Effects of the punchboard as evidenced by a test 
different in type from the practice tests. The above tC 
sults were obtained with immediate repetition of the 
same test. Conceivably, learning brought about by the 
punchboard might be highly specific—perhaps a mere 
memorizing of the test material, or the key. If this were 
true, a second test different from the practice matena 
would show little gain. The following simple experiment 
offers evidence that the learning was not so limited, that 
there was instead a real gain in knowledge of the matter 
presented. 

The multiple-choice Russian test used previously was 
first given a class, using the punchboard. Then a very 
different test, a recall test, was given. On this recall test, 
the Russian words were listed in a different order from 
that in which they appeared on the practice test, and a 
space was provided after each Russian word for the stu- 
dent to write its English meaning. This same recall test 


what occurred simply with repetition of the test as a test, of 
only about 5 per cent. The big effect of the punchboard was 
in correction of errors. The punchboard brought immediate 
right answer on second trial for about 65 pér cent of item 
‘wrong on the first testing. 


| 


— SV 
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was given also to another similar class, after the practice 
test had been taken simply as a test. However, the order 
of items was different. And the “direction” of the think- 
ing was reversed. In the check test, a definition was pre 
sented and the student indicated which of four difficult 
English words fitted the definition. In contrast, the prac- 
tice test presented the word and the student chose the 
correct one of four definitions. The Psychology check test 
differed from the practice test in an even more elaborate 
fashion. Ten questions on one general topic did not ap- 
pear in any form on the practice sheet; 10 were para- 
phrased from questions on the practice test; the remain- 
ing 10 were repeated from the practice series but ap- 
peared in a different sequence on the 
with the paraphrased and new q : 
order of the wrong answers and positi e right 
answers changed. This scheme was use! 
in this study. Its purpose was to obtain evidence regard- 
ing the spread of punchboard learning as affected Y 
degree of similarity, in phrasing AY idea, between 
the practice and the check tests. Table 2 $ 
Tesults of these check tests, given immediately after the 
corresponding practice test had been taken wit 
Punchboard. 


Effects e here not so great ot SO 
of the punchboard are here nov ely the same 


clearcut as 
in Table l; a Te etition 
we would inevitably show T nore gain than check tests 
Engin from the instructional test. $ 
ish and psychology the differences ar stantia! 
l (as indicated by the note to Table 2) distinct. 
Sb of the four-point gain in P 
on the “re-arranged repeat” items a poi 
on the rashla analysis here of little reliability 
gate to be further investigated. 
Usin; Effects of several consecutiv 
wile Big punchboard. If once through cP 
i e punchboard does pring some learn, e 
times through should bring Mor. However, if students 
80 through the same selEinstructional test S 


1 . 
leapa Scession, it is conceivable that they might 


tn the key or pattern of right answers rath} j 
information conveyed in the test. Or at Teast, they might 
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TABLE 2 


Learning from taking a test with the punchboard, as shown 
by number of errors on a second test different in form. 


Subject matter of | oe s fa è 
iste | Russian English Psychology 
Method of giving | As Punch- | As Punch-| Punch- 

first test: | Test board | Test board | Test board 
Median number of 

errors on second 

test 19 17 9 2 10° 6 


2 Only 5 per cent of the class taking the English practice test 
simply as a test, and 12 per cent of those so taking the psychology 
practice test, did as well as or better than the median of the punch- 
board group. Table 3 following shows further reduction of Russian 
recall test errors to 10 after 5 punchboard testings. This is addi- 
tional evidence that the practice tests were causing a learning 
which was not confined to mere rote learning of the practice test 
series. Again, these classes averaged 30 students in size. 


learn that information so much in one particular order 
and context that use in other setting would be handi- 
capped. To check on these points, one class was given the 
same Russian vocabulary test practice test four times 
using the punchboard, and then a recall check test, al 
in one class hour. A similar class was then given a series 
of practice tests covering the same words but with the 
order of items and the key or pattern of right answers 
changed each time; again the punchboards were used but 
changed each time to fit the key for that particular test. 
And again at the end of the series the recall and test was 
given. Similar series were run with same and change 
tests in English vocabulary and psychology except that 
the length of the hour permitted only three times 
through the practice series, and left no time for an €n 
test in psychology. Table 3 shows the median number 
A o on each practice test and in the Russian chec 
est. 

On the Russian, the changing practice tests seem actu- 
ally to handicap the learning, in that the median num er 
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¥ errors actually goes up on the second trial, and there- 
after continues higher than on the tests in which the 


TABLE 3 


Perene in number of errors with repeated taking of a practice test 
sing the punchboard, and effects of changing the order of questions 


and the pattern of right answers upon such decrease.” 


Russian Vocabulary English Psychology 
Practice Practice Practice 
Test Check | Test Check | Test 


12345 Test 1 23 Test T es 

a Eee a Sal 
Median No. 
of errors 

Same 

test 11 853 10 | 42145 6 17 4 1 
Changed 

315996 10 41178 5 18 3 1 


li a Numbers of cases for Russian groups were 
ish 29 and 41, and for psychology 33 and 34. Error scores on en 
tests are mistakes plus omissions. Five minutes study of 

ractice tests. 


sian vocabulary list preceded the p! 


order was unchanged. However, the check test shows n° 
difference, at the end of the practice series. On the Eng- 
lish vocabulary and psychology, results for same and 
changed practice material are practically the same. | 

A natural interpretation of the above results 1s that 
with such réte learning as of the Russian, there tends to 
be some dependence on the arrangement of items. How- 
ever, in proportion to the meaningfulness and integration 
of the material, change Of sameness Of key makes no 
i and that 1s 


difference; attention is on the subject-matter, a 
ith none of these subject- 


what is learned. Apparently wit h 
aiid does repetition of the practice test with the same 
ey handicap learning. It is € ijable that the 
Same test and key in $ ight help. For 
example, in reviewing 4 book it is a 
each time find a topic on the same page ¥ co 
text, and upsetting if a review is of a different edition 
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where all these details are different. The analogy is of | 
course only very rough, but it does seem suggestive of 
value in review of the same test. Some value would seem 
especially likely if the test questions were in a logical 
order and perhaps grouped under headings, as suggested 

a few pages back. Anyhow, these findings are indeed 
pleasant, since they support the more convenient repeti- 
tion of the same test for practice. They also thus warrant 
use of a specialized device, for facilitating repeated runs 
through a practice test, to be mentioned later.? 

In short, the punchboard does indeed appear to further 
learning. It is probably more effective with meaningful 
than rôte matter. Repeated use seems to carry learning 
further; and some repetition of the same test does not 
seem to cause mere learning of a particular test key or 
other unfortunate effect. However, the results so far have 
been with brief and relatively simple tests and checks on 
outcomes, all within one class hour. The practical ques- 
tion now is whether these procedures would be effective 
with more substantial learning tasks, and over a period 
of time. 


2. The Punchboard, in a College Course, as 
a Testing Aid Not Integrated with Teaching 


The most simple but obviously inadequate way to use 
a device like the punchboard is as a convenience in test- 
ing but without any attempt to tie its instructional po- 
tentialities in with teaching methods. Such a situation 
might occur if its use were required of teachers without 
interesting them in it or indicating ways of using it to 
aid instruction. Two trials of what might be called per- 
functory use were made. 

a. Use of punchboard tests for review. In the large 
second-quarter course in general psychology, eight chap- 
ters of the textbook were covered between the first an 
second midterm examinations. To help students in 1e- 
, 7 Readers familiar with the methods of Thorndike and others 
in the study of the law of effect will recognize that repetition 
of a test using the punchboard involves the “retained situation, 
the problem being “retained” before the learner until the 
right answer is found. It would seem that such devices as the 
Punchboard could be very useful in such research. Results © 
such theoretical investigation will be reported in a later pape! 


an — 
. a 
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viewi Sees 

ig for the second examination, two 60-item review 

paee constructed. The first consisted of 20 multiple- 
e questions on the first two chapters, 20 on the 


second two, and 20 more which were the first 20 repeated 


but in a different i i 
f ; _ order. This review test was giver to 
es totaling 114 students during the das hour 
alil gme midterm examination. A second review test 
ae sie covering the second four chapters was given to 
lain r wo totaling 182 students. On the day fol- 
net all sections were given an “jective midterm 
ee having 21 questions repeated or para- 
fet rom the first review test, 22 similarly from the 
nd practice test, and 24 questions which were on the 


sa i e : y : 
Tee eight chapters but unlike questions on either review 


P, TABLE 4 

es r F r A 

EERE correct, of mid-term questions covered in a review practice 
using the punchboard, and of other questions not thus reviewed. 


Group A | Group B 
| ae 


aa 
Questi i 
ions not in practice tests 16% 15% 
85 69 


Questions in practice test given Group A 
70 79 


Questions in practice test given Group B 


Table 4 shows that on these new questions the two 


groups did practically the same, thus indicating that they 
a ability and accomplishment 


were essentially equal in 

aside from any help the prac ight have given. 
On the midterm questions which were repeated or para: 
E tased from the first practice test the group which had 
ae it did distinctly betters similarly, the second group 
dif etter on questions from the e 
in «ahi: are not great, 
ae oe that use of the P 
aa as repeated questions. 
t hose which had been rephrased 
o the midterm. 

In the above experiment, no significant differences 


ed errors most 
in appeared also 
from the practice test 
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appeared between questions which appeared once and 
twice on the practice test. Initial difficulty was found 
very important, however. Obviously, the punchboard 
could bring no more than 5 per cent improvement on a 
question so easy that 95 per cent of the students at once 
answered it correctly on the practice test. On 1ost of the 
difficult items, the device caused large gains. However, 


TABLE 5 


Per cents of students getting questions right on midterm which they 
(a) had taken on a practice test using the punchboard, and 
(b) had not taken on a practice test. 


Not Taken | Gain From 
Taken in With Punch- 
Practice | Punch- board 
Test board Test 
Questions paraphrased from 
practice test to midterm 69 72 3 
Questions repeated from prac- 
tice test to midterm 69 87 18 


The table shows that 72 per cent of the students got questions 
right on the midterm which they had had in different wording on 
a practice test using the punchboard. However, only 69 per cent 
of students got these questions right if they had not taken them 
in the practice test. The punchboard practice test thus brought 
a gain of 3 per cent; in repeated questions the gain was 18 per cent. 


on a few paraphrased questions, the practice test actually 
caused loss, presumably because of tricky rephrasing. 

b. Value of frequent brief practice tests using the 
punchboard. In the next quarter, trial was made of the 
possible value of frequent brief testing using the punch- 
board, in the second month of the course in general psy- 
chology, in which eight chapters of the test were covered- 
Four 20-question practice tests were made up, each sam- 
pling points covered in two chapters. On the day when 
discussion of one of these two chapter units was to begin, 
five sections totaling 149 students took the practice test 
on that unit using the punchboard. Afterward, any ques- 
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tions about the test were answered; but since the instruc- 
o regarded the testing as an experiment in which they 
oe no hand, they did not encourage discussion nor use 
k e test in their plan of teaching. Five other sections, 
otaling 152 students, on the same days took these same 
practice tests in the usual fashion of test-taking; they 
amply marked on the test sheet what they thought to be 
ahi right answers. The papers were then collected, graded 
a er class with all errors marked, and returned the next 
day. At this time the instructor went over the questions, 
indicating the right answers, and giving opportunity for 
discussion. This procedure might be considered to lean 
over backwards in the effort to assure full demonstration 
of the values of frequent test without the punchboard. 
A total of more class time was given to the tests than to 
the punchboard procedure. The tests were seen twice, on 


two different days. Probably in most classes in which fre- 
isnt brief tests are given, the scored tests are returned 
ess promptly, and without such review of questions ani 


tight answers. A third group of five sections covered the 
same ground, but without taking the practice tests in any 
way. Most of the instructors had two sections; and as far 
as possible it was arranged that each had one of one type 


and one of another. 

Evidence as to the initial status of the three sets each 

of five sections before the experiment began was yie'de 
ination, since all 15 sections 


by the first midterm examinatio | - 
used the same instructional procedure during this first 
month of the course. The first row of figures in Table 6, 
giving the average per cent of right answers made by each 

i the “no practice 


group on this first examin 

test” group initially best and the punchboard group 
poorest. On the second midterm the two groups which 
have taken the practice tests are slightly above the first 
group. Analysis shows the punchboard group best on the 


25 questions re eated and 15 para 
p 3 uestions—as might be 


tice tests, but poorest on new q 
anticipated bot their initial bottom position (Table 6). 
_ Occasional brief tests not closely tied in with instruc- 
tional methods thus brought only slight gains. Tests 


given in the ordinary Way, scored after class, then re- 
turned and discussed the next day, were about as effective 
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TABLE 6 


Percentages of questions answered correctly by three groups of students 
(a) on the first midterm, before the experiment began, (b) on the 
second midterm, and (c) on each type of question in the second 


midterms 
Groups of Students 
Tests 
No Tests merce 
Practice Taken i 
Test. With 
ests as 
Punch- 
Taken Test a 
First midterm 69 68 66 
Second midterm 73 78 76 
28 questions not in the prac- 
tice tests in any form 73 75 69 
15 paraphrased questions 77 82 83 
25 repeated from practice 
tests 71 81 83 


* Five sections in each group, the first totaling 149 students, 
the second 152, and the third 149, 


as the punchboard. However, it did save the work of 
scoring, also the trouble and time of returning answer 
slips the next day and going over test questions and right 
answers with the class then. But it would seem that tests 
should not be merely occasional episodes incidental to 
the instruction, but rather be closely integrated with the 
instruction method. What might the gains then be? 


3. Use of Practice Tests With the Punchboard 
as a Major Instructional Method, by the Instructor 
in Charge of the Class 


This experimentation was in the course in educational 
psychology, which follows one in general psychology, 15 
most commoniy taken in the freshman year, is taught in 
sections of about 35 students, and meets five times 4 
week for one class hour. For this work, a large number of 
Practice tests had been carefully prepared, with special 
emphasis on problems of judgment and application. 


f 
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a. Value of intensive use of punchboard tests as com- 
pared with carefully planned discussion by the same in- 
structors. First, what might be accomplished by using the 
punchboard as a major instructional device, on certain 
units of the course? These units or topics (as, transfer 
of training) were in the course calendar allotted two or 
three class hours. On the first day assigned to such a 
unit, the experimental sections took a 30-item practice 
test using the punchboard, at the beginning of the hour. 
When a student finished the test, he looked up any 
doubtful questions, and discussed these with other stu- 
dents and the instructor. Since it was understood that 
the practice tests would not be used directly as a basis 
for grading, but rather as an aid toward preparation for 
midterms and final examinations, both instructor and stu- 
dent sought help from the practice testing. A student’s 
total classroom work, including his attitude and effective- 


ness in use of the practice tests, did figure in the larger 
appraisal of him by the instructor, however. After issues 
i d of, further dis- 


raised by the practice tests were disposed 
cussion took such direction as the situation to date sug- 
gested. This was to fill any: gaps remaining in considera- 
tion of the subject, and to generalize and apply | i 
tial as a safeguard against he undue specificity whic 

the objective study tests might otherwise foster. Then 
near the end of the second day assigned to a topic) a 
second practice test was given, with a little time follow- 
ing that for discussion. Finally, the day after the topic 
Was finished, a 30-item check test was given in whig i 

questions were repeated from one or the other of the 


ice ] e total 
Practice tests, 10 paraphrased, and 10 new- Of the to! 

topics o the course, Six cattered through the quarter 
died ae few minutes for the 


were handled in this manner. : i 
check test thus always came from @ unit not given the 
special treatment. ; 
The above procedure was used in two experiments) 
Sections taught by two experienced instructors W a an 
ad a control section, where as much th Ey t 
o discussion as was allotted t the 0 
ion. In their control sections, 1 
to stress points which had comé out in the punchbeag 
groups or apply other helpful ori 
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the experimental procedures; except that the punchboards 
and practice tests were not used on the six units. The 
remaining 12 topics of the course were taught by infor- 
mal discussion in all sections. 

Selection of the two experimental and two control 
sections, from the total of all sections, was by chance 
circumstance. Their status at the beginning of the ex- 
periment was determined by an index taking account of 
each student’s standing on the test of general ability 
given at entrance, his academic record to date, his grade 
in the preceding course in general psychology, and his 
Score on a pre-test given at the beginning of the educa- 
tional psychology course and anticipating the major 
topics of that course, to measure pre-knowledge. The 
experimental and control groups were found essentially 
equal initially, as well illustrated by the pre-test means 
in Table 8 to be presented shortly. 

The first analysis was of the effects of the experimental 
Procedures on the check tests given the day after each 
unit. Table 7 shows the percentages of questions correct 
in each group for all six check tests combined, or a total 
of 60 questions in the check tests which had not ap- 
peared in any form in the practice tests, 60 paraphrased 
for the practice tests, and 60 repeated from them. Analy- 
sis of variance indicated that in total the differences even 
on the “new items” between experimental and control 
gtoups could be considered significant. It appears that 
when such a diagnostic test-teach device is used as an 
integral part of instruction, results are substantial, and 
they seem to bring better understanding of the topic as 
a whole, so that even “new” questions in that topic are 
more often answered correctly. 

It might be considered that the check tests were given 
too soon after a topic had been completed, and were too 
closely tied in with the practice tests. Table 8 shows the 
tesults of tests given later, having no such tie-in, and 
providing much broader coverage. In total, these covere 
all 18 topics in the course, were made up independently 
of the practice tests, and included an extensive case study 
test, unlike all the other material. Pre-test scores illustrate 
findings with a measure of initial status. Means on the 
pre-test are practically the same for control and punch- 


| 


tet, 


i 
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TABLE 7 


epee of questions correct on end tests, in two classes using 

ae chboard tests as a regular part of instructional procedure, and in 

control groups taught by the same instructors which took no 
practice tests.° 


No Punch. Punch- 
Practice ee í board 
Test sa Gain 


60 Questions not in practice 
tests 54 


60 Questions paraphrased from| 
practice tests 58 75 17 
60 Questions repeated from 
60 83 23 


practice tests 


d a The two sections using no practice tests had a total of 57 stu- 
ents and the punchboard sections 62. 


board groups, but on the midterms plus final the punch- 
board group is clearly superior, the difference surpassing 


the 1 per cent level of confidence. , 
As in the appraisal of the review test 1m general psy- 
chology reported earlier, the design of the above expen- 
ment combined values bf the punchboard and the prac- 
tice tests. The combination was here also justified on the 
ated the frequent testing 


ground that the device so facilit 
ight be here included as part of 


that values of the tests mg ) 
the total value of the punchboard. However, evidence 
was again desired which would differentiate the effective- 
ness of the practice tests with and without the punch- 


© As already mentioned, two instructors were involved in the 

h teaching one control and one experi- 
mental section, so that any differences 10 their effectiveness 
might affect both gro f the means of the 
groups taught by the first instructor and the second showed no 
significant difference. The infer s that on, the whole the 
two instructors did not substan jally differ in the groups they 
had to work with, or_in their efficiency. Analysis of variance 
also showed no significant difference between ‘the instructors 
in the relations between their control and experimental groups; 
the practice tests with punchboards seem to have been used 
by them with about the same effectiveness. 
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board. The set-up now to be described was planned to 
take account of this issue. 

b. Value of frequent brief tests with and without 
the punchboard. The plan was simple and straightfor- 
ward. In the spring quarter there were 13 sections of 
educational psychology. Four of these used practice tests 
with the punchboard in connection with certain units of 
the course. They were taught by the two instructors who 
had just completed the project described above and thus 


TABLE 8 


Average scores made by 2 sections who had not taken any practice 

tests and 2 other sections who had taken practice tests on 6 out of 18 

topics, on (a) a pre-test covering essential background and major 

points given at the beginning of the quarter and (b) two objective 

midterms and a final examination given during and at the end of 
the quarler.* 


No Punch- 
Practice board | Difference 
Tests Tests 
Mean score on pre-test 45.2 45.6 4 
Mean of midterms plus final 286.7 298.8 12.1 


* There were 57 students for whom there were complete records 
in the first group, and 62 in the punchboard group. Sigma for the 
total midterm plus final score of the first group was 18.9 and of 
the second 16.6. 


were accustomed to the use of the punchboard. Two 
sections took the same practice tests on these units, but 
instead of using the punchboards they marked their 
answers in the usual fashion on answer sheets. As they 
finished, they discussed the questions among themselves 
and with the instructor, thus arriving at knowledge of 
the right answers and clarifying doubtful points. The 
tests were then collected, and the answer slips grade 

and returned ‘a day or so later. As before, this method 
probably leaned over backward in an effort to give fu 

opportunity for advantages of the practice tests without 
the punchboard to be obtained; rather more time was 
given to test-taking plus discussion of the tests than with 
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punchboard group, and return of the answer slips was a 


second reminder of the test and opportunity for discus- 


sion of them. Seven sections did not use these practice 
tests in any way, the time going instead for informal dis- 
cussion. Initial status was again 
an aggregate index taking account of ability as te 
entrance to the University, scholarship to date, and grade 
“a the preceding course in general psychology. Again the 
1 Tee groups were 
lowances for initial differences were needed. 

During the first four weeks of the quarter, five units OF 


topics of the course were covered. The punchboard 


groups began consideration of two of these with a punch- 
int of depar- 


board practice test which was used as a point 
ture for discussion. They then took a second practice 
test on the second day, in the same fashion as the punch- 
board groups in the previous project. The test groups 
took the practice tests on these units in the usual paper- 


and-pencil fashion. The remaining sections covered these 
on. And this last 


two topics entirely by informal discussi 


method was used by all 13 sections to cover the remain 
ing three units. A 75-question multiple-choice midterm 


Was then given all sections. Of these questions, 25 were 
Tepeated or paraphrased from the practice tests, 10 were 
new questions but on the same two dealt with 
in the practice tests, and i the three 
Cee not having practice test 
he second four weeks of the course four U 
sidered, on two of which two practice tes 
the punchboard and test groups: The second midterm 
was then given. It had 24 items repeated 
from the practice tests, 16 jons on the same 
units, and the remaining 35 on the two topics which were 
new to all groups. 
h The bars in Fig. 1 show the score’ ma 
alf of the students in the three groups» 
midterms. The total scores from such v y 
questions are obviously the undifferentiated sum © 
Ous possible effects of the practice tests. Bul 
worth considering as evid 6 what might 
well become a common situation: practi 
some but not all the material in @ course, 


de by the middle 
on these two 
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differentiating portions so aided. The differences between 
the no-practice-test group and the punchboard groups are 
evident and clearly significant. Superiority of the punch- 
board sections over the sections taking the practice test 


FIRST MIDTERM SECOND MIDTERM 


tions which appraise students’ work as a whole without i 
No Pr. As Punch- No Pr, As Punch- 


s Tost Test board Test Test board ¢ 
c 
£ 7 a 
& e | 
60 60 | 
55 55 
50 50. 
45 45 
Maan < f 
40 40 


Fig. 1. Scores made of two midterms by three groups of stu- 
dents, those who had taken (a) no practice tests (b) practice 
rere simply as tests, and te} practice tests using the punch- 

oard. 

There were 225 students in the seven sections not taking the 
practice tests, 70 in the two sections taking them simply as tests 
but with discussion after, and 132 in the four sections taking the 
practice tests with the punchboard. The bars show scores made 
by the middle half of the students in each group (from the 2 
to the 75 percentiles). The practice tests covered only about 
half of all the topics included in the midterms. 


without the punchboard is evident, especially on the sec- 
ond midterm. As always, individual differences in a 
groups are great. 

Analyses of results are shown, in Table 9, for all the 
questions dealing with the units covered by the practice 
tests, and 10 items in each midterm on topics not Ss? 
covered. The first three columns of figures show average 
Per cents for each group passing each type of item on the 
midterm; the fourth column shows the gain of the punci- 
board group over the non-test control, and the last col 


a 
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TABLE 9 
Percentages of students who had taken (a) no practice tests, (b) practice 


tests simply as tests, and (c) practice tests with the punchboard* 


who 


answered correctly various types of midterm questions. 


Relation of Midterm to 
Practice Tests 


First Midterm: 

10 Questions not in the prac- 
tice tests in any form, and 
on different topics 

10 Questions not in practice. 
tests in any form but on 
same large topics 

0 Items paraphrased from 
practice tests 

S Items repeated from the 
practice tests 


Second Midterm: 

10 Questions not in practice 
tests in any form and on 

Fi different topics 

6 Questions not in practice 
tests in any form, but on 
same topics 

14 Questions paraphrased 
from practice test 

Questions repeated from 


i Punchboard 
Practice Tests Cane Over 
Not As Punch-| No As 
Taken Test board | Test Test 
Per Cent Correct 
63 ól 68 5 7 
52 60 62 10 2 
66 74 77 11 3 
77 90 91 14 1 
Per Cent Correct 
78 30 83 Cm N 
65 72 78 13 6 
6o 74 76 8 2 
3 %4 89 16 5 


i ° There were 225 students in the 7 se 
ice tests, 70 in the 2 sections taking the 
iscussion after, and 132 in the 4 section: 


with the punchboard. 


umn the gain of the pun 


control” sections. 


The fourth column shows 
Punchboard group over the group W; 
tests. On repeat and paraphrased items, 


ctions not taking the prac- 
m simply as tests but with 
s taking the practice tests 


chboard sections over the “test 


substantial gains of the 
p which took no practice 
the gains are 
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greater than those for use of punchboard not integrated 


with instruction presented earlier in Table 6. In addition, 
the punchboard sections show superiority on new ques- 
tions dealing with the same units, and even on questions 
on the units not covered by the practice tests. Apparently 
the discussion and teaching tied in with the tests, and 
aiming at generalization and application, had so oper- 
ated. Moreover, the right-hand or fifth column shows 
some superiority of the punchboard over the test-control 
group; and this extends also, on both midterms, to ques- 
tions on topics not covered by the practice tests. To what 
extent these last differences may be considered significant 
is not clear, but their appearance in all categories surely 
warrants added confidence in them. And again it must be 
remembered that punchboard testing was also more con- 
venient, time-saving, and interesting to the students than 
testne in conventional fashion with discussion the next 
ay.? 


, ° The question might be asked: “Were the punchboard sec- 
tion instructors better”? Their “control” sections the previous 
quarter had been clearly below their own punchboard sections 
and at the average of other instructors; the quarter before that, 
before the experiment began, their sections had been no better 
than the others on midterms and finals. Might superiorities 0! 
both test groups over the control be due merely to greater habi- 
tuation to tests? Present-day college students are used to tests; 10 
the previous course in general psychology they are regularly used: 
Better scores on questions repeated or paraphrased from tne 
practice tests would be expected, from students who had taken 
them, but should not be wholly discounted as due to undesirable 
rôte recall. After all, it is good instruction to bring up important 
points in class discussion, and sound testing that these points 
should also be covered in any later tests. And the gains on topics 
not covered by the practice tests remained to be explained. 
Apparently the punchboard tests, when integrated with instruc 
tion, brought more adequate coverage and generalizing of su?” 
ject-matter than usual instruction—more even than the less 
convenient testing without them. i 

Doubtless there should have been a greater range of practice 
and appraising test material. But the experimentation include? 
judgment tests, and a unique case study test. Earlier investiga: 
tion under the writer's direction, with a basically similar devic 
for self-instructional testing, had used true-false practice tests, 
appraised outcomes with multiple-choice and essay-type ma 
tions, and obtained similar positive results (Little, 1934). ; d: 
et. the range in materials and methodology seemed cons! 
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c. Student attitudes toward the punchboard. The 


above findings were the results of objective testing. Indi- 
gens of students’ attitudes toward the punchboard 
Ng re sought. Their liking for it was well evidenced by 

eae ‘protest when, after its use part of a quarter, the 
aoe design called for its discon 
poe the device sufficiently helpful that they came to 

pend on its immediate appraisa 
jonny thereafter, tests taken 
leaving them uninformed in 


tinuance. They 


Is, and guidance to right 
in the usual way, and 
these respects, were frus- 
unities were given to take a test 


her punchboard tests on the same 


topic (several were available on each) many students did 


thus extend their self-instruction. Punchboard tests were 
e both to students and instructors, 
were invited. The 
to attitudes, and 


trating. When opport 


et such use. Comments from students 
ollowing samples indicate something as 
methods of using. 
Every time I punch a wrong answer, I read the ques 
over again, The punchboard makes me think. 
When I have the right answer, but am. not certain, the 
on that answer and 


punchboard assures me; then I can settle on that $ 
remember it, and consider the reasons why it 1s tight. 
Direct questions help me I don’t retain as much from 


general discussion. : 
My errors are pointed out immediately; I’m not left with 
the wrong ideas. 

New ideas and angles are brought out and underscored. 

It’s like a game and a challenge; I'm motivated more. 
_ If you punch a wrong hole, you're the only one who knows 
it; it’s not upsetting in class discussion. 

With the series of questions and all the time finding out 
how much I know, I can concentrate much better than in 


ordinary study. 

After I’ve been through a punchboard test, I feel much 
more confident, know where I stand, have been checked up. 
I find it much easier to ask for help, since the punchboard 


brings issues to a head. 


tion 


In short, use of an occasional punchboaid test some- 
what perfunctorily as 4 convenience in testing brought a 
little specific gain in learning. However, use which was 
tied in with instructional method brought decidedly 
more, and spread that gain 50 that general understand- 
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ings were bettered and study methods made more effec- 
tive. Further, under these circumstances, students liked 
the device and appreciated it as helpful. 


4. Use of Punchboard Practice Tests with Special 
Groups of Superior Students 


It might be assumed that a device facilitating imme- 
diate understanding of test material and self-instruction 
would be most effectively used by the ablest students. 
Since a class, like a fleet, tends to travel at the rate of the 
slower members, any scheme which releases the abler 
from that restraint should be valuable. Three methods of 
using the punchboard tests with groups of superior stu- 
dents were therefore tried. 

a. The “Accelerate Seminar.” An accelerate seminar 
was tried four different quarters. Selection of students for 
the seminar was based on a brief student data sheet filled 
out the first day of the course in educational psychology. 
This sheet covered age, academic record, veteran status, 
part-time employment, and interest in the seminar as 
briefly described to the group. Those students who were 
interested, whose academic records and scores on the 
entrance test of general ability were high, who were not 
over-burdened with part-time employment, and who 
seemed to have a reasonable participation in student 
social life and activities, were then interviewed; and the 
most promising were chosen for this special group. Avet- 
age number in the seminars was 26, or a bit less than 4 
tenth of the total 300 or so students taking the course 
each quarter. Average percentile of the seminar group 07 
the test of ability at entrance was 77 and academic avet- 
age slightly above B. A majority but not all were veterans. 

The seminar met only two hours a week (usually in the 
evening) instead of the usual five one-hour sessions. 
the regular readings and projects were covered and in ac 
dition about a third more reading, to compensate for less 
class time and assure against possible criticism that the 
seminars were too easy. Three or four punchboard tests 
on every topic, most with page references to the reading 
after each question to aid systematic check back on wea 
points, were made available to the students in a file, an 
they were shown how to load the punchboards with paper 
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hee if so that they could do everything themselves. 
aa Ags ise use the tests for self-appraisal and guid- 
Fis at ne l-group discussion, at certain class times and 
open rag: ae hours in the afternoon when the file was 
opel em. Part of the class seminar hour was given 
Oiri projects or other undertakings and to informal 
Eae A at a mature seminar” level. And since class 
aa = saved, each student was supposed to take an 
ies ae urse, work part-time, OT make other good use of 
was aoe e T e aE, Median course load 
‘ele s, and median hours of part-time employ- 
ee Scie outcome, students were given essay €x- 
aie ions upon the extra reading, were graded by the 
E on projects and participation in class discus- 
on took the two objective midterms and final given 
E o the regular sections. The first rows of Table 10 
ti mmarize the results in terms of grades on these objec- 
pars made by the regular sections, the accelerate 
eminar students, and cases in the regular sections paired 


with the seminar cases as to 28° sex, point-hour-ratio 
through the previous quarter, and grade in the previous 
course in general psychology- ith all these categories, 
pairing was a bit rough, but was reasonably satisfactory- 

i e over twice as 


High (A and B) grades in seminar wer 
peo proportionally as in the regular sections, an 
ailing (E) grades rare. Moreover, the seminar students 
who had over twice 


were better than the paired controls, 
were more intangible gains; 


many seminar students spoke of increased capacity for 
i ith other stu- 


ak pendent work, and closer relations wit 
ents and the instructor because 
t be 


et as easily as one regular se 
n number of students 
were involved. But it seemed cle: 


tests, could save class fm 
total of college work- Moreover, 
course did not handicap elsewhere; 


the demands of this 
a follow-up ° these 
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TABLE 10 


Grades made by superior students using special instructional methods 
including punchboard tests, as compared with grades of students using 
conventional methods of instruction. 


Percentage of 
Students Obtaining | Number 
A and B t 
Grades Ee Students 
27 regular sections 28 11 801 
4 accelerate seminars 65 2 106 
Cases paired with seminar cases 55 5 106 
2 examination-credit groups 96 0 24 
Cases paired with above cases 83 0 24 
2 self-instructional laboratories 67 0 24 


. Taaa a = S an 


° A and B grades are high, but Æ is failing. Grades here given 
were based on two objectives midterm tests plus an objective 
examination including a case study test. 


students and their control groups about a year later 
showed the seminar group slightly more superior in aca- 
demic record than before. And as a result of an extra 
course at the time of the seminar plus extra credit 0 
tained at some other time, a considerable number 0 
these students graduated a quarter or more sooner than 
would have been possible otherwise. F a 
b. The examination credit group. Might superior an 
well-motivated students be put yet more on their ow? 
with yet more guidance and help from the self-instruc- 
tional tests? Two “examination credit” groups gave 4” 
affirmative answer to this question. University tules pi 
vide that a student may obtain “examination” credit be 
a course without taking it, if he obtains a grade of B ob 
better on an examination covering the course. The haat 
tion was as to whether superior students could, if ne a 
a little guidance in independent study plus the help oa 3 
series of punchboard tests covering the course, pass ee 
educational psychology without any regular class her 
_ For this little investigation, records to date of ne 
in the pre-requisite general elementary psychology we 


E 


> 


” 
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Tun over before the middle of the quarter, to find stu- 
dents doing exceptionally well in that course and other- 
wise superior. The best of those thus discovered were 
interviewed to verify as to their superiority, explain the 
plan, and see if they were interested. Only those strongly 
interested and clearly superior were finally included. 
Average percentile on the entrance test of general ability 
was 92, average grade to date was high B (a PHR of 3.35) 
and average age 20.5; preference was given to veterans. 
One quarter, nine students were chosen, and another D: 

As soon as each group was selected, near the middle of 
the quarter, the first of a series of four two-hour meetings 
was held. In these, procedure was further explained, as- 
signments made in large blocks each covering about a 


third of the course, certain laboratory projects briefly 
demonstrated, and two objective midterm tests and a 
final given; there was time for a very little orienting dis- 
cussion. In addition to the four meetings, the total series 
of punchboard tests was made available during about four 
hours each week with at least two hours’ attendance Te- 
quired. An additional 20 hours each week was designated 
hen an instructor OT assistant was avail- 
able for consultation and punchboard tests were also 
available. Work was thus largely independent study but 
with frequent punchboard test check, and opportunity for 
individual consultation. Often two or more students were 


in for the check tests at the same time, and worked 


together. 


In the row labelled “examination credit group” of the 


Its are shown in terms of objective mid- 
given regular sections also. The following 


row gives results for cases paired with those in this group 
as to ability and previous work, in the same fashion as 
for the accelerate seminars. Again results are excellent as 
compared with regular sections, but not quite as good 
as for the paired cases. Perhaps this indicates the students 
were approaching the limits as to desirable proportion of 
independent work. à 

1 Neither accelerate seminar nor proficiency group were 
without precedent. Worcester (1945), Munroe (1926), and 
Umstattd (1935) have reported successful trial of special sec- 
tions meeting less time a week, for superior students, in several 
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c. The self-instructional laboratory. A third venture 
attempted even greater flexibility. At the beginning of 
two different quarters, small groups of very superior and 
mature students were given the opportunity to take the 
course in educational psychology in a “self-instructional 
laboratory.” Each student was given a list of all the work 
in the course, including the usual informal laboratory 
exercises. Again a third more reading was asked of the 
special groups. Five afternoons a week, a special labora- 
tory room was made available to these students, in which 
all the materials and readings for the course and a com- 
plete series of punchboard tests covering the course were 
provided. In charge was an assistant who was an experi- 
enced teacher and thoroughly familiar with the course 
and the practice tests. The students were told that there 
would be no regular class meetings. They could come into 
the laboratory whenever they wished, work with others 
or not, consult when they wished with the assistant. Each 
could progress at his own rate, test his understanding of 
each topic as he finished work on it, easily look up any 
item on which he had trouble (using the page references 
after each question), and then check himself again with 
a different punchboard test. When each student felt 
ready for a midterm or the final, he could ask for it. Each 
could thus finish the course at his own rate; all finished 
by the middle of the quarter. Results are shown in the 
last row of the table above. Over two-thirds of these 
superior and mature students made A or B grades. Some 
used the time saved in part-time employment, others 10 
taking extra courses or yet other ways. Several reported 
gains in effectiveness in independent work. 

In short, self-instructional tests made it possible. for 
superior students to work largely or even almost entirely 
independently in various types of special groups. Muc! 
time was thus saved both instructors and students, and 
students’ programs either enriched or completed sooner 
than otherwise. 


different subjects. The University of Buffalo (Mills, 1936) 
pioneered in trial of intensive preparation for proficiency exam 
nations. All the above studies reported favorable outcomes: 
The distinctive feature of the work at Ohio State University was 
the systematic use of the self-scoring instructional tests. 
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D. MAJOR CONCLUSIONS 


The preceding pages have reported a variety of proj- 
ects. What generalizations from them seem most impor- 
tant? Four major conclusions appear to stand out from 
the total investigation. 

1. It has demonstrated a simple way to telescope into 
one single simultaneous process the taking of a test, the 
scoring of it, the informing of students as to their errors, 
and their guidance to the finding of the right answers. As 
stressed at the beginning of this paper, the usual test 
must be scored after the class meeting when it is taken. 
If the students are to know their scores an 
errors were, the errors mus 
and marked sheet and test ques 
If they are to correct their errors, they 
what the right answers are. All these steps 


nated at one sweep, by self-scoring devices. Not only is 
time and trouble save 


d. Since usual test scores are not 
available until a later day, any use of those scores must 
wait till then. Self-scoring devices make possible immedi- 
ate action. Especially in any situation where expeditious- 
ness is highly important (as in a mass training program 
in a national emergency } t 

ble, and elimination of delays in use, could cumulatively 


be of decided importance- Irrespective of any self-instruc- 
tional values, simple self-scoring devices should thus be 
worthwhile as 4 means for saving time and labor, and 


speeding up the total testing process: 

i 2. The F vestigation has shown that such a telescoped 
testing process, which informs each student immediately 
ch question whether his answer is correct, 


as he answers ea 
and guides him to the right answer when he is wrong, 
does indeed transform test-taking into a form of system- 
atically directed self-instruction. Immediate outcomes of 
such testing with the punchboard appeared amply to 
demonstrate that this was So. Repetition of a self-instruc- 
tional test brought marked reduction in number of errors 
made. These effects showed also on tests of different 
types than the instructional tests. The learning was pti- 


marily of the subject matter and not of the test key. 
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However, use of the self-scoring device in a college course 
as a somewhat incidental feature in occasional brief tests, 
brought only limited gains in scores on later objective 
course examinations; and these gains were confined 
largely to specific items covered in the practice tests. As 
with other teaching aids, self-instructional tests must be 
made a carefully planned part of the total instructional 
program, if their values are to be adequately realized. 
Their occasional incidental use must not be expected to 
show their real potentialities. 

3. The investigation has shown that when the self- 
instructional tests were used systematically in college 
courses as an integral part of the-teaching method, gains 
were substantial, and sufficiently generalized to improve 
understanding of a topic as a whole—even help on related 
topics. Thus used, the “punchboard” tests brought better 
work in regular classes, as shown by higher scores on mid- 
term tests and final examinations (even though used in 
only about a third of the course) than made by com 
parable sections of the same course not employing the 
punchboard. The device was found especially valuable 
with superior students. Sections of such students, using 4 
series of sclf-instructional tests systematically covering a 
course, did superior work with saving of 60 per cent or 
more in number of class hours. There was thus marked 
reduction in amount of instruction needed, and the stu- 
dents saved sufficient time so that they could take extra 
work, many finishing their education sooner in conse- 
quence. 

It appeared that substantial use of carefully planned 
comprehensive series of instructional tests could have ad- 
vantages, over the same time in discussion, somewhat 
comparable to the advantages of an objective test over an 
essay type examination or an oral quiz. More questions 
could be asked, in a given time, of every student, cover’ 
ing a topic more systematically and adequately. The right 
answers in the tests could be very carefully phrased to be 
adequate. The wrong answers could bring up and then 
dispose of common misconceptions. Such helps as pase 
references after each question could guide review. The 
self-scoring device brings all these values immediately an 
incisively. Both the experimental findings and the very 
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iabe attitude of students suggest that all this was so. 
of In school or college, such materials should be of value 
eae ge aids in regular classes, and of special help 
pee groups. In the Services, particularly in a na- 
‘se Nc game the immediate scoring plus self-instruc- 
oe a hould accelerate training programs. Special groups 
tai ficer training might cover more, subjects or complete 
3 eee New training materials sent out to ships 
ie o distant ports might be distinctly more effective, 
if accompanied by self-instructional tests. Programs such 
as fostered by the United States Armed Forces Institute 
might gain in both interest and effectiveness, if such tests 
were included. 

4. This paper reports results with the punchboard. It 
was found simple and convenient even for student self- 
use, and could readily be made yet more compact and 
convenient. However, the total project has shown that 

ising means for automatic scoring 
and self-instruction. A “chemocard” was tried. It was 
three by five inches and looked much like the face of 


the punchboard. Across t 
dent’s name, the date, and like in! 
lowed two columns of 15 rows of four circles, the rows 
being numbered from 1 to 30. It could thus be used with 
30-question, four-choice_ tests. Each student was given 
a cheap fountain pen filled with a special red ink with 
which he wrote his name and other informa 
top. He then marked through the first circle of the first 
row if he considered the first alternative for the first ques- 
tion to be right. If it was, the mark immediately turned 
gray (as a result of an invisible chemical printed in the 
“right” circle); if the mark did not turn, the student 
tried another circle in the first row, continuing until the 
right answer was found. Total errors was the total of 
unchanged red marks. The chemo-card obviously operates 
to test and teach in the same way as the punchboard, but 


the materials are simpler.* : 


u It is believed possible to print right circles with an invisible 
chemical such that any pen and ink can be used and the tight 
answer will at once blur as in soft blotting paper. The idea 


of an invisible chemical to indicate correctness of response was 
developed by the writer's former student, the late Dr. Hans 
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A key machine developed on the basis of earlier work 
(Pressey, 1926) not only indicates errors and requires 
finding of rights before continuing, but also keeps a 
cumulative count of errors and can readily be used for 
repeated runs through the test to reach mastery. Yet 
another mechanism, again based on earlier work (Pressey, 
1927), adds to the above features in that on successive 
tuns through a test, only those questions are returned 
for further drill on which a mistake was made the previ- 
ous time through. Such selective review according to each 
student’s need should be yet more effective.12 

In short, “human engineering” can aid educational 
and training programs by test-teach devices of various 
types. The major purpose of this project has been to evi- 
dence the value of the basic idea, as illustrated by the 
punchboard, and determine ways of using such a device 
so as substantially to improve instruction or training. The 
value of such devices, and the need for carefully planned 
methods for their use, if those values are to be realized, 
both seem clear. Research aiming still more to realize 
these values, and to appraise certain of the other devices 
mentioned above, is now under way. 
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Knowledge of Results in 
Self-Teaching Spelling* 


J. Wituram Moore AND WENDELL I. SMITH 
Bucknell University 


Three experiments utilizing auto-instructional mate- 
rials were undertaken to obtain additional evidence on 
the use of self-teaching materials. The first two studies 
to be reported dea] with the assumption that information 
on the correctness of each response must be provided to 
the student with a minimal delay (Lumsdaine, 1960). 

In previous studies, immediate “knowledge of results” 
(KR) has been provided in a number of ways. In some 
cases, S is informed of a correct response by some me- 
chanical or electrical device, such as a buzzer or a light. 
In other cases, particularly when the response required 
of S is based on direct recall rather than recognition, the 
desired answer is given to provide S with a standard 
against which to compare his response. In this latter 
type, both self-teaching textbooks and teaching machines 
have been used as the media for the presentation of self- 
instructional materials. i 

Tt is claimed that by utilizing self-teaching materials, 
a student may proceed at his own rate, and that he is in 
no way restricted by the rate at which his peers learn the 
same material. As a corollary it is claimed that most stu- 


* This work was supported, in part, by the Susquehanna 
Valley Program, Bucknell University, through a grant from the 
Ford Foundation. This article is reprinted from Psychological 
Reports, 1961, 9, 717-726, and it is used here with the permis- 
sion of the authors and the editor. 
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dents can learn a greater amount when using auto- 
instructional materials ad libitum than they can in “lock- 
ee instruction (Blyth, 1960). The question explored 
y the third study to be reported is, how is the rate of 
presentation of materials related to the acquisition and 
retention of the information presented? 


EXPERIMENT I 


Although there is some disagreement on the best 
method to use for providing S with knowledge of the 
accuracy of his response, most programmers accept the 
assumption that knowledge of the correct response with 
a minimum of delay is necessary for effective learning to 
take place (Skinner, 1958). A test of this assumption is 
made in this experiment. 


Procedure: Two classes of sixth-grade students (N= 
62) participated in the study. The classes, which met 
in separate buildings, contained 28 and 34 Ss respectively. 
The range of 1.Q.’s (Otis Form A) in the group was 
67-123, with a mean of 100.42. To reduce the effects of 
her variability Ss were randomly assigned to 
] groups within each classroom. 


gnment there were 15 girls and 
and 17 girls and 12 


experimental and contro’ 

As a result of random assi 

18 boys in the experimental group, 

boys in the control group. s 

Fifteen units of programmed spelling were provided 

ing the last fifteen weeks of the 1959-60 

school year. To test the effects of providing for S the 

relatively short delay each mem- 

used, in addition to the 
a small answer ‘booklet from whic 

e correct response to a frame within 

is response. The control 

ed units without the an- 


programmed units, 
he could obtain th 
a few seconds after writing hi 
group used the same programm 
swer booklet. 

The teachers? were instructed to provide, each S with 


1The self-teaching materials were developed by Douglas 


Porter, Harvard University. 
2Mr. Guy Long and Mr. Bernard Zabarowski, instructors in 
the Danville Area schools, were the teachers who participated 


in the experiment. 
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a programmed unit on Monday morning during the regu- 


larly scheduled spelling period. The experimental group ` 


was given the answer booklets containing the correct re- 
sponses and they were told to check the accuracy of their 
answer immediately after it had been written. Students 
were allowed 15 minutes to work on the unit; at the end 
of this time the units were collected. The same pro- 
cedure was followed on Tuesday. No other instruction in 
spelling was given. On Wednesday of each week Ss were 
given an orally administered test on the 17 words in- 
cluded in the unit. All Ss were given the same word list 
used in the Wednesday test to study on Thursday. On 
Friday, they were tested again on this word list. This 
procedure was followed throughout the 15 week period. 

All Ss were pre-tested and post-tested with the Metro- 
politan Spelling Achievement Test, Form R within a 
week prior to and following the experimental period. 

Results: It was hypothesized that providing immediate 
knowledge of the correct response might increase the 
amount retained over a period of weeks. If this were 
true, it would be evidenced in a gain in spelling achieve- 
ment between pre- and post-testing. Differences between 
the groups on the Wednesday test might be attributable 
to the use or non-use of the answer booklets. Differences 
in gains between the Wednesday and Friday test scores 
might be attributable to something akin to reminiscence, 
or it may be that mode of response interacted with the 
Thursday review of the word list. 

Mean gains between pre- and post-testing were com- 
pared for the experimental and the control groups within 
each classroom and for two classrooms combined. The 
results are shown in Table 1. 

It can be observed from Table 1 that the control 
groups were consistently higher in spelling achievement 
than the experimental groups; however, in no case was 
the difference significant. y 

In comparing the means for the two experimental and 
the two control groups and the combined experimental 
and control groups on Wednesday test scores, on only 
one unit was the difference significant at the .05 Jevel. In 
this case, it was in favor of the control group. Neither the 
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TABLE 1, EXPERIMENT I 
Comparison of gains on metropolitan spelling achievement test. 


Mean Gain | Mean Gain 
School Control Exp. b, — 
(N = 29) (N = 33) 


A 2.00 1.43 oe 

_B 5.57 3.35 1.18 
Combined groups 

(A + B) 3.79 2.48 1.06 


experimentals nor the controls were consistently higher 
in mean achievement score on each of the units, 

In comparing the mean gains on each unit for the 
| experimental and the control groups between Wednesday 
_ and Friday scores, significant differences were obtained 
in only one instance. In this case, the experimental group 
was significantly higher than the control group. A com- 
_ parison of the mean gains between Wednesday and 
_ Friday test scores on the’ seventeen units combined 
again resulted in a t which failed to reach the .05 level 
of significance. 

It could be concluded from this study that providing 
Ss with KR from an answer booklet (the independent 
variable) did not aid the learning of spelling. However, 
a number of factors which may have reduced the meas- 
urable effects of the independent variable are worthy of 
consideration. First, the mean number of words spelled 
correctly on the Wednesday tests was slightly more than 
15 in both groups. When this is compared with the 
mean number of words (17) contained in each spelling 
$ unit, it is apparent that there were not enough words 
per unit and/or the level of difficulty of the words was 
not great enough to discriminate between the slow and 
tapid learners. 

The failure to attain a significant difference when using 
the Metropolitan Spelling Achievement Test as a_meas- 
ure of retention may have been the result of the follow- 
S. 4ng: (1) all children were given the word list to study on 
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Thursday of each week, possibly reducing the measurable 
difference in the post-tests; and (2), although the Metro- “ 
politan Spelling Achievement Test may have some valid- 
ity as a criterion measure in this experiment, a test com-_ 
posed of words taken directly from the units would have 
greater validity. 

Since the programmed units were used in a text form, 
cheating may have been a problem. Specifically, it is 
probable that in a few cases Ss checked the correct 
response before writing the response to the item, thereby 
ane the hypothesized effectiveness of the immediate 


EXPERIMENT II . ‘ 


Since the results of the first experiment were equivocal, 
a second study, in which more adequate controls were 
provided, was undertaken. ‘ 


Procedure: A class of 28 sixth-grade students wer 
chosen as Ss for the experiment. To approximate th 
population of Experiment I as nearly as possible, this 
study was initiated one year later in one of the same 
classrooms with one of the same teachers as in the first 
experiment. The Ss attending the school were from the 
same geographic area as were the students of the pre- 
ceeding year. 

The 28 sixth-grade students were randomly assigned ' 
to the experimental and control groups with equal num- 
bers of boys and girls in each group. The chronological 
age ranged from 11 years 2 months to 14 years 3 months, 
with a mean age of 12 years 1 month. The 1.0.’s (Otis _ 
Form B) for the group ranged from 81-128 with a mean / 
of 107.31. After random assignment of the groups the 
mean I.Q.’s of the two groups were compared by a t-test 
analysis. The means did not differ significantly at the 
05 level. =~ 

A spelling test consisting of the 400 words from the 
original spelling list was administered to the Ss approx 
mately three months before the experiment began. From = 


š 
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a. 
` this test the 210 words missed most frequently were 
~ selected for inclusion in a revised form of the programmed 
"units. To increase the level of difficulty of the task 
further, the number of words included in each unit was 
doubled. In the units used in Experiment I the number 
-of words per unit was 17; in the revised units, the 
number of words varied from 32 to 39 per unit. A: a 
result, six new units were formed, four of which were 
used in the four week experiment. 

A pre-test, composed of a random selection of fifty 
words from the revised programmed units was given to 
the Ss one day before the beginning of the experimental 
period. 

The programmed units were presented to Ss in a 
Fosinger Teaching Machine (write-in type) to eliminate 
copying of answers. For the experimental group, the 
correct answers appeared in the window of the machine 
when the lever was pulled. The control group used the 
I ines and programmed units but did not- 
of the correct answer. This was 
jaccomplished by placing masking tape over the window 

f the machine where the correct answer normally ap- 


rocedure was as follows: On 
f the first week, the experimental 
ith the machines. On Wednesday and 
i placed on the machines 


and the control group completed the unt 
‘was necessary since only fifteen mac 
While one group wo 
the other group worked quietly on other subject matter. 
A maximum of twenty minutes 
terials was allowed on each of the two days. Th 


| vided sufficient time for the slowest Ss; for the brighter Ss, 


= =i) 


time than was required. Upon completion 
the specified time had not been used com- 


_ ments in other subjects. For t 
dure was reversed, i.e., the control group comp 
unit on Monday and Tuesday and the experimental group 


4 completed the unit on Wednesday and Thursday. The 
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iocedure of alternating the machines was continue 
throughout the four weeks of the experimental period. No 
other instructions were given to either group except thost 
necessary for proper operation of the teaching machines 
On Friday of each week all Ss were given a spelling 
test consisting of the words contained in the unit as- 
signed for the week. For each test, the teacher pro- 
nounced each word, used it in a sentence, and pronounced 
it again. | 
On the Monday following the four-week experimental 
period, a post-test was given to both groups. It was com- | 
posed of fifty words selected randomly from the pro- 
grammed units. ; 
Results: Means for each weekly test, the post-test, and 
combined weekly tests were compared for the experi 
mental and the control groups. The results are shown i 


Table 1, 


. TABLE 1, EXPERIMENT II A 
t. 
Comparison of mean scores on unit achievement measures y 


in spelling. 
Exp. Control 
Number Group Group 
Test of Mean No. | Mean No. t 
Words Correct Correct 
r (N = 14) (N = 14) 5 
omna 
Test 1 32 words 28.0 29.78 a9 g 
Test 2 37 words 32.214 35.0 1.217 
Test 3 33 words 27.286 30.357 151 
Test 4 39 words 30.214 34.286 1.367 
Post-test | 50 words 41.286 43.786 0.727 
Total 191 words 159.0 173.429 a ly 


It can be observed from Table 1 that the contro 

oup was consistently higher in spelling achievemen 
than was the experimental group, however, in no instanc 
was the difference in scores significant. 
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ISCUSSION 
q Since the results from neither experiment demonstrated 
ignificant differences between experimental and control 
y be concluded that providing S with knowl- 
ate his learn- 


‘groups, it may 
edge of the correct response does not facilit 


ing of spelling. 
Given a literal interpreta 
that the effectiveness © self: instructional materials in 


spelling (Porter, 195 
of the material rather tha 
providing immediate KX. This is not to imply that some 
KR was not available (and a ne y 
learning) for the control group. It must be remembere 
did not receive im- 


that although the control group 
ing the correct answer after they had 


' i rediate KR by seeing 
% sponded, the words were used in later items. 
in KR, though the delay in rein- 


Wala, therefore, obtal 
eG l€ ment was Jonger- F A 
pts CElVis of some interest to note that in both experiments 
ccor means On the criterio e€ consistently 
f ther (though not signific roups which 
pea! not have immediate i 
l; correct response to an item before movin) 
ext item in which a similar response i 
ghe a case of over-prompting- at is, the hypothesized 
Tpeneficial aspects of KR in the form use 
may in fact reduce learning © i when the appear 
Vance of the correct answer serv 
item. Specifically over-cueing ma) 
by reducing learning 


, 
articipation of the student, there 


‘efficiency ; 
i Although the data do not provide direct support for 
i iate knowledge of the correct response, 


K 
\the use of imme! 
irable attributes of programmed material 
i | First, both the experi- 


h 
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= 
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| mental and 
. level of achieve 


room teacher. 
the disadvantage of over-cueing OF prompt- 


2 Some support for 
Goldbeck (1960). 


| 
t 
ing is given by 
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and post-test data were available, the mean number offen 
students who got each word correct was computed for o 
each test. The mean and S.D. for the pre-test was X = 
15.33, standard deviation = 4.22, and the mean and sD 
for the post-test was X = 23.06, standard deviation = 
2.649. Second, the same number of words (as indicated 
in the first experiment) were learned by the students as 
in a traditional method to the same criterion level, with 
a saving of at least fifty per cent of time for all students. 
The bright students had even greater savings. Third, the 
teacher was released almost entirely from the respon- 
sibilities of dealing with the formal teaching of spelling, 
thereby allowing more time for other instructional activi- 
ties. 

In summary, these experiments support some of the 
claims which have been made for programmed materials 
in a classroom, but they question the need for providing 
KR as operationally defined in this experiment. It is 
suggested that the advantage of KR may be nothing more 
than an additional cue for the response to the next item. 
If this is the case, and the conventional KR method is 
to be used, then the number of cues within the items" 


must be Teduced; otherwise the response may become 
eal with little learning taking place (Goldbeck, 


-ata 


On the other hand, the intrinsic value of KR may be of 
Teal worth as most authorities in the field of automated 
mstruction have suggested. If this is true, then more 


research must be conducted to differentiate between the 
reinforce 


ee ment properties and the prompting properties 


EXPERIMENT II $ 


. There is little evidence resulting from current research 
automated instructi 


i ir on concerning the rate of presenta- 
m of natria] to the student as it effects acquisi- 
and retention of the material presented. How- 
“Coulson and Silberman (19 % 

3 59) made som t E 

measure ret i i e E f 
naar bi A air but did not relate it to the rate of presenta- 
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ever, the results of studies (eg. Kimble, 1949; Hull, 
1943) investigating spaced vs. massed practice would 
seem to offer some evidence in favor of a controlled 
presentation of auto-instructional materials. A generaliza- 
tion of this type can be made only on the assumption that 
the use of programmed materials ad libitum and massed 
learning have in common some of the same elements 
which effect learning. 

The third study was done to compare the effects of 
completing limited and unlimited amounts of auto-in- 
structional materials, in a specified period of time, on the 
c= Sia and retention of spelling by sixth-grade stu- 

ents. à 


Procedure: A class of 35 sixth-grade students was 
chosen to be Ss in the experiment. The Ss were ran- 
uomly assigned to experimental and control groups. 
There were 10 males and 8 females in the control group 
and 10 males and 7 females in the experimental group. 
The I.Q.’s ranged from 78 to 131 with a mean of 110.52. 
The mean I.Q.’s of the two groups were compared by a 
t-test analysis and did not differ significantly. 

To compare the effects of (a) completing programmed 
units as rapidly as possible and (b) limiting the number 
of programmed units which could be completed in a 
designated period of time, the following procedure was 
used: Six units of spelling were programmed into a self- 
teaching textbook form. The method for selecting the 
words and developing the units was described in Ex- 
periment II. The same materials were used by the experi- 
mental and the control groups. Both groups also were 
provided with answer booklets. 

The control group completed one unit each week. They 
were allowed twenty minutes each Tuesday and Thursday 
for the completion of the unit. If S did not require 
the entire forty minutes to complete the unit, he devoted 
the remainder of his time to other subjects. On Friday 
Ss were given fifteen minutes to complete®a self-admin- 
istering spelling test. The test was made up of sentences 
using the words they had used in the unit during the 
week. Upon completion of the test, the date on which the 
test was taken was recorded on the top of their paper. 
This procedure was used over a six-week period. 
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To measure the retention of spelling in the control 
group, the test which was given at the end of each unit 
was readministered six weeks later. A post-test consisting 
of a random selection of words which had appeared in all 
six units was given nine wecks after the last unit was 
completed. 

The Ss in the experimental group were permitted to 
complete the programmed units ad libitum. They were 
given fifty-five minutes a week to work on units or to 
take tests on the units. The major difference between 
the groups was that an S in the experimental group com- 
pleted as many units as possible in the allotted time 
while the Ss in the control group were restricted to one 
unit per week. If an S in the experimental group com- 
pieted a unit in one period, he was permitted to take 
the self-administering test at the end of the period if 
time was available; if not, he took it at the beginning of 
the next period. He could then proceed to the next unit. 
This procedure made it possible for some Ss to complete 
two or more units and tests during each week. When 
each unit test was completed by an S in the experimental 
group, he recorded the date at the top of the paper. To 
measure retention, the same test for each unit was read- 
ministered exactly six weeks after each of the first tests. 
The same post-test which was adnrinistered to the control 
group was given to the experimental group exactly nine 
weeks after the mean time it took the group to complete 
all six units. 

Results: The mean achievement scores for the spelling 
test administered immediately after the units had been 
completed for the experimental and control groups were 
compared by the t-test analysis. The means of the groups 
and the results of the statistical comparisons are given 
in Table 1. 

In no case did the difference between means approach 
the required level of significance. 

A t-test analysis was used to compare the loss in 
achievement (as a measure of retention) between test I 
and test II of each unit and the difference between the 
scores made on the post-test by the experimental and 
control groups. Table 2 presents the data. 
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TABLE 1, EXPERIMENT II 
Comparison of mean scores on achievement measure in spelling. 


Unit I Exp. Mean | Control Mean t 
I 28.94 28.47 38 
Ir 28.39 27.29 85 
Il 34.26 34.06 _ 17 
IV 31.06 30.75 95 
Vv 34.22 34.81 39 
36.82 35.38 83 


TABLE 2, EXPERIMENT IM 
Comparison of losses in achievement on unit spelling tests. 


š Exp. Control 

Tak Z-Z% | %-X i 
I 1.24 2.00 0.25 
II 1.14 2.77 0.32 
sat 1.88 1.33 0.15 
IV 0.53 1.80 1.46 
v 0.94 2.25 1.22 
1.19 1.81 0.18 


DISCUSSION 


The data from this study do not provide any evidence 
that massed practice with programmed materials affects 
retention adversely. The failure to obtain differences 
may be a function of the kind of programmed material 
used in this study, or.to the relatively short lapse of time 
between criterion measures used. This study lends some 
support to those who recommend that the student pro- 
ceed at his own rate when using programmed materials. 

One advantage of presenting material to Ss ad libitum 
was the saving of time. The experimentalegroup spent a 
mean working time of only fourteen school days on the 
six units, while the control group spent the entire eight- 
een days.5 Further, the group receiving material ad 
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libitum appeared to be more highly motivated than the 
control group. This supports Skinner’s views on the 
motivational properties of automated instruction (Skin- 
ner, 1958). 
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6 Actually, the two groups spent approximately the same 
amount of time on the units, for the control group, after 
finishing a unit early, worked on other materials. However, 
under conventional teaching conditions the class is kept to- 
gether, regulated by the “average student” in the class. 
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Teaching Machines: An Investigation 
of Constructed Versus Multiple- 
Choice Methods of Response* 


Epwarp B. Fry 
Loyola University, Los Angeles 


_ Teaching machines are defined as automatic or par- 
tially automatic devices which present a question or other 
stimulus to a student, provide a means of response, and 
then inform him of the correctness of his response im- 
mediately after he has responded. 

An exhaustive survey of the related literature revealed 
that Pressey issued the first published report of a teaching 
machine in 1926. His was a simple mechanical device 
making use of multiple-choice questions. Later he and 
his students developed studies using punchboards that 
gave immediate knowledge of results to multiple-choice 
questions. Pressey believed the use of teaching machines 
to be in harmony with such learning theory as that pro- 
posed by Thorndike. 

In 1954 Skinner proposed that teaching machines were 
effective and in harmony with such major learning prin- 
ciples as “successive approximations” and an adequate 
schedule of reinforcement (frequent rewards after small 
units of work). Skinner and his followers expressed strong 


* This article is reprinted from Automated Teaching Bulle- 
tin, 1959, 1, 11-12, and is used here with the permission of 
the author and the editor. 
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preference for devices which required that the student 
respond by constructing his own answer, for example, by 
writing the answer from memory rather than by recog- 
nizing and selecting one of several proffered choices. 

The purpose of this study was to determine, if possible, 
which of the two modes of response was the more efficient 
for teaching a list of Spanish words and phrases to 153 
ninth grade beginning Spanish students. A pilot study 
showed that when students were required to construct 
Tesponses, better scores resulted on a post test but that 
a longer training time was taken. 

The major portion of the study was conducted under 
several controlled conditions in which the variables of 
time and number of repetitions could be studied. Under 
Condition I, both experimental groups (multiple-choice 
versus constructed response) worked to a mastery criterion 
of two correct responses while working time was re- 
corded. Under Condition II, both groups were stopped 
before subject mastery was reached, thus allowing them 
equal total working time. Under Condition IH, the 
stimulus items (English words) were presented to both 
groups by means of a large flashcard, thus allowing equal 
stimulus presentation time and equal responding time 
for each item besides controlling the number of responses 
made by each student. Learning was measured during 
two post-tests, both--consisting of equal numbers of 
multiple-choice and constructed items. One post-test was 
given immediately after training, the other after a two- 
day interval. 

Results: Responses to the multiple-choice items all 
approximated the maximum possible score and hence 
did not reflect any significant differences. Constructed 
responses, on the other hand, showed significant results 
favoring the constructed mode of training response under 
all three conditions for both post-tests. Under Condition 
I, where time was allowed to vary, constructed responses 
took significantly longer. 

Conclusions: Given the conditions prevailing in this 
study, constructed training responses result in more learn- 
ing than do multiple-choice responses if the criterion of 
learning is recall. The literature has indicated that the 
use of teaching machines offers great promise in the 
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teaching of many phases of nearly every school subject. 
Teaching machines enable one to make interesting ap- 
plication of learning theory and the results of learning 
experiments, and would appear to merit considerable 
attention from educators. They are excellent research tools 
for curriculum construction and psychological experimen- 
tation. 

Where it is required that the student be able to recall 
the learning material unaided, and given the conditions 
of this experiment, constructed training responses should 
be used, even if training time must be limited. 
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~ An Analysis of Response Mode and 
Feedback Factors in Automated 
Instruction* 


Rosert A. GOLDBECK AND LESLIE J. Briccs 
American Institute for Research 


INTRODUCTION 


This paper discusses a variety of issues arising in the 
psychology of learning as applied in automated instruction. 
As in earlier conventional learning tasks employed in 
experimental psychology, discussion of continuous dis- 
course programs of the kind pioneered by Skinner ( 1958) 
conveniently can center around factors relating to (a) 
stimulus conditions, (b) response characteristics, and 
(c) knowledge of results, or feedback. Temporal and 
sequence factors involved in variations in these three sets 
of conditions become a focal point for much of the dis- 
cussion of learning problems and issues. However, added 
richness in variation of conditions and sequences of 
events appears to characterize continuous discourse learn- 
ing, as compared, for example, to paired-associate rote 
learning. This added richness brings increased complexity 
to attempted functional analysis of the learning process. 
And due to the nature of continuous discourse material 
and current methods of programming the learning, more 


* This article is reprinted from Technical Report No. 2 of 
the American Institute of Research, and it is reprinted here 
with the permission of the author and the Institute. 
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complex sequences of stimulus, response, and feedback 
events characterize the learner’s activity for a single item 
or frame. Thus, the analysis of such learning does not 
rest complete with mere recognition of a written single 
stimulus term, an overt response, and a simple feedback — 
signal, in that order. 4 
Past conceptualizations of paired-asgociate learning, 
though extremely complex in the theoretical or inter- 
pretative aspect of the learing, in functional analysis 
tended to consist of rather simple recognition of an 
S-R-Reinforcement sequence for each pair of terms. 
Since the concept of stimulus supports, now prominent 
in continuous discourse material, was not present in 
conventional paired-associate material, and since the brief 
stimulus term was constant throughout the trials, overt 
responses and reinforcement of them figured as the promi- 
nent factors in leaming theory. Also, as the exposure 
of both stimulus and reinforcement terms was rigidly 
paced, the subject was presumed to be forced to be en- 
aged, for each item, in reacting in tum to the S, the R, 
and the feedback term. This simple, rigid assumption, of 
course, ignores the responding processes actually taking 
place covertly in the subject, and gives an over-simplified 
appearance of orderly knowledge concerning learning as a 
function of the overt response and reinforcement. Thus, 
learning came to be carefully plotted out in terms of 
response frequency as learning trials progress. Some rich- 
ness was introduced into theory by considering interpreta- 
tions of such factors as similarity, overt intrusions, and 
other sources of variation in learning speed for single 
items, but in general, the notion arose that learning con- 
sists of making overt responses and receiving reinforce- 
ment. 
It is the view of the writers that the above conception 
of learning is by no means adequate in understanding of 
or arranging for the learning of continuous discourse, 
conceptual material. And, even though this paper deals 
with responding and reinforcement variabtes, the writers 
view the stimulus conditions and the implicit and covert 
behavior of the learner as prime factors in their tentative 
conceptualization of the role of the selected learning 


components discussed herein. 
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This paper is in part a reporting of some of the 
Tationale development growing out of the first major 
experiment under the present contract. In this respect, 
many of the comments expand upon the reasons for 
conducting that experiment; other comments grow out 
of interpretations of results obtained (Goldbeck, 1960). 
In addition, this paper presents some of ‘the views of the 
writers on matters which they believe require additional 
research to achieve further clarification and understanding 
of the factors in programming for effective learning. In 
this context, much of the discussion is intended to refer 
to procedures for programming of continuus discourse 
material along the general lines of Skinner’s methods. 
However, the content of the paper is more generally 
intended to call attention to basic issues in the psychol- 
ogy of learning in order to enlist interest by others in 
further research and discussion of issues. 

Initial learning, or response acquisition is the primary 
emphasis in the paper, although some of the comments 
are relevant to retention, transfer, and use of the learning. 

The experimental study cited above, and this paper 
both represent considerable focus upon two factors in 
learning the writers consider of major importance. First, 
our emphasis upon implicit and covert responding departs 
somewhat from the more behavioral language of papers 
which place relatively greater emphasis on the overt 
performance of the subject. Second, our viewpoint on 
item difficulty and the effects of right and wrong responses 
differs from the view that learning cannot be made too 
easy: Our view is based upon our concept of issues of 
cueing and prompting, learning versus performance, im- 
plicit versus overt responding, and upon empirical data. 

Regarding terminology, it is well to note that we 
distinguish “response mode” from “response pattern.” 
By response mode, we mean the rather formal aspect of 
responding, such as multiple-choice versus constructed 
response, or overt versus covert. By responding, or response 
pattern, we refer to the total implicit and overt aspects 
of reacting, the former being not directly observed. 
Although experimental investigation of some of the learn- 
ing factors we discuss would yield data which would 
aid in inferring the nature of covert activity, it appears 
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that methods and techniques for measuring effects of 
these processes more directly are to be sought. 

It is hoped that this paper will help focus attention on 
some of the many still elusive conditions of effective 
learning, so that further experimentation and communica- 
tion may result in better understanding of and program- 
ming for effective learning. 


AUTOMATED INSTRUCTION AND EDUCATION 


The development of automated teaching techniques 
and the wide interest they have stimulated among both 
learning theorists and educators have contributed sub- 
stantially toward bridging a gap that has long existed 
between the problems of learning studied in the labora- 
tory and the problems of teaching in the classroom. This 
previous lack of a beneficial interaction of effort can be 
ascribed mainly to a difference in goals; these differing 
goals in turn prescribe a difference in both subject matter 
and method. 

The primary goals of the learning theorist have been 
concerned with developing knowledge about the relation- 
ships among the variables of learning. Subject matter 
and methods have been selected especially to best achieve 
these goals. The goals of the educator have been con- 
cerned with achieving the most effective learning of sub- 
ject matter chosen on the basis of educational needs. 
The methods selected have been those which were be- 
lieved to produce the best results, judging from classroom 
experience. j 

Now that some of the results from the laboratory have 
been channelized into methods for effectively teaching 
classroom subject matter, in the form of programmed 
instruction devices, there has been established a common 
ground of interest’ and effort among experimentalists 
‘and educators. But, educators are still concerned with 
some goals that have not been encompassed by pro- 

rammed instruction techniques. These techniques are 
not intended to supplant the teacher and all existing 
instructional methods, but to serve as an effective aid 
to the educational process. It is to be hoped, however, 
that the problems of the educator will continue to moti- 
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vate the development of programming techniques that 
will free the teacher to play a more productive role in 
the educational process. The contributions of the educator 
should be of considerable value in both the advancement 
of programming and the incorporation of programmed 
instruction into the educational system. 


AVENUES FOR CONTINUED IMPROVEMENTS 
IN PROGRAMMING 


The development of improved programmed instruc- 
tion techniques can proceed in several different ways. On 
the one hand, it is recognized that the student and 
teacher view the learning procedure as an integrated 
process. Since the scientists, too, must be aware that 
the various aspects or variables involved in the learning 
process have their impact on the learning outcome in a 
combined fashion, instructional methods may be studied 
in a holistic fashion. It is thus possible to conceive of 
and develop a complete programmed learning procedure 
and compare it with some other proposed procedure in 
an effort to arrive at conclusions concerning the “best” 
method of instruction. Such an effort depends heavily on 
the ingenuity of the developer. It is quite possible that 
the most effective instructional methods will be derived 
in this manner. Certainly, eventual adoption and use of 
instructional methods should depend on some such 
comparison of complete methods. 

On the other hand, it should be apparent that the 
development of “complete methods” benefits from the 
theoretical analysis of the learning process into its com- 
ponents and the empirical evidence from studies which 
deal with these components at a relatively specific level. 
Auto-instructional methods and programs already pre- 
pared are largely based on such previous analyses and 
experiments. Modifications and improvements of existing 
techniques will benefit from further analysis and experi- 
ments conceived within the framework established by 
these recently developed techniques. Studies of interac- 
tions of single factors in various technique combinations 
are an important aspect of such research. The improve- 
ment may come from an emphasis on those features 
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which are found to contribute heavily to the learning 
results or from data suggesting de-emphasis and perhaps 
elimination of those features which do not contribute to 
learning success and which may, in fact, degrade the 
learning process. Findings which lead to new concepts 
and, hence, to new versions of techniques are an even 
more important contribution that can result. 


THE FOCI OF NEEDED STUDY 


There are four major focal points of the learning proc- 
ess as it occurs with programmed instruction: 


1. Learning material presented 

2. Response required 

3. Feedback supplied 

4. More learning material presented 


There are many variations possible in the operations 
specified for the student to perform at each of these 
points, and variations in the combinations of operations. 
Specifying a particular set of operations defines the 
characteristics of the auto-instructional program to be 
used. There are currently two contrasting points of view 
concerning what the characteristics of programming 
should be. 

Skinner and his followers hold that precise control 
should be maintained over the student’s pattern of re- 
sponding so that desired responses can be shaped by a 
process of careful reinforcement in some successive ap- 
proximation sequence of learning frames. It is done in 
a fashion which avoids rote learning and provides under- 
standing by eliciting the desired stimulus-response con- 
nection under a variety of conditions. 

Pressey, Crowder, and others hold that the student’s 
pattern of responding should be stimulated to the limits 
of his capability. This concept of programmed instruction 
emphasizes the thinking or less controlled and more 
implicit responses that occur during learning. Feedback 
is provided for multiple-choice responses’ to reinforce 
and augment successful learning, to redirect the study 
process, if necessary, and to correct errors. 

The many possible variations in the four focal points 
of programmed instruction and their combinations, each 
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representing a particular instructional technique, involve 
responses and feedback which are covert and uncontrolled 
to some degree, whether they lean toward the shaping 
concept or the “thinking” concept. 

A major task which the writers believe to be necessary 
is to untangle the specific variables that have been as- 
sociated with particular instructional techniques and to 
relate these variables to learning outcomes. It is believed 
that such relationships depend rather heavily on the 
covert and implicit activities of the student and that the 
delineation and manipulation of variables should be 
undertaken with a recognition that active participation 
of the student is not restricted to reading what is pre- 
sented to him and writing symbols or words in the 
spaces provided. 


RESPONSE MAKING AND FEEDBACK 


The remainder of the present discussion will be 
concerned primarily with points (2) and (3)—response 
making and feedback. But, as will be apparent throughout 
this paper, consideration of the effective use of these 
variables is closely associated with the form of the learn- 
ing material, especially in terms of the cueing or prompt- 
ing characteristics. One outstanding characteristic of 
most programmed material is its orientation to the stu- 
dent and his learning characteristics. Material has been 
written so that the student will be able to follow and 
understand each item of information as it is presented. 
This orientation has been maintained by using the 
criterion of student response as the guiding principle for 
preparation and revision of material. This emphasis on 
successful student response in preparing material, aside 
from the use of response making and feedback during 
learning, may be credited for a substantial portion of 
me success that programmed instruction has achieved to 

ate. 

A consideration of how response making and feedback 
should be used in programmed instruction to produce the 
desired learning outcome requires some delineation of the 
characteristics and descriptive dimensions of response 
and feedback. These characteristics can then be examined 
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for the ways in which they may be employed to contribute 
to the learning outcome. It should be emphasized again, 
however, that the operationally defined conditions of 
responding and feedback may have implications for 
producing student behavior in addition to the activities 
explicitly required by the program. These may -be media- 
tors of the learning required or they may have such 
independent effects as keeping up effort, increasing con- 
centration, maintaining alertness, etc. For work proposed 
in 1961, one goal is development of improved methods 
for study of implicit responses. Some of this behavior can 
be observed and measured if appropriate conditions are 
employed. For example, study of eye movements of sub- 
jects should help estimate time used in sequences of 
actions such as reading, re-reading, thinking, and response 
writing. Eye movement studies have been made to un- 
derstand and improve reading, and to learn how subjects 
scan a work of art. Records of such movements could be 
useful to show scanning of stimulus materials, viewing 
feedback terms, glancing back at written answers, etc. 
With appropriate apparatus, this could be a valuable 
form of investigation under controlled or paced lessons, 
as well as for study of individual differences in a less 
structured program performance. Sub-vocal throat activity 
also might be useful to supplement other methods of 
observation and measurement. 

Certain aspects of implicit behavior may not be 
amenable to direct observational techniques and must be 
inferred from behavior which can be measured. The stu- 
dent’s own implicit responses may range far beyond the 
stimulus materials, reaching back in time to make new 
associations and forward in time, possibly in anticipation 
of the next frame, the test to follow, or in making inven- 
tive responses not measured in terms of the criteria or 
learning outcomes planned by the programmer. Arous- 
ing such desirable, implicit activity may be more signifi- 
cant in reaching the defined criterion than are overt 
perfunctory responses. Also, these implicit activities may 
be useful in invention, transfer, etc. 


Methods for Producing Desired Responses: Control of 
behavior, it would appear, has been approached in the 
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past through many different methods and techniques. The 
methods used in psychology have changed through the 
years, and they do, without doubt, vary in effectiveness. 
The methods used also have historically varied somewhat 
systematically with the nature of the material to be 
learned. The theory of learning developed has reflected 
the methods and materials used in the attempts to control 
behavior by experimental conditions. Some of the typical 
methods, and resulting emphases in’ theory, may be 
summarized as follows: 

1. The Confirmation Method. This method, typically 
used in memory drum experiments, presented a stimulus 
(S) and required the subject to attempt to make a re- 
sponse (R), after which the correct response term was 
exposed. No hint or cue beyond the “bare” S was given. 
On the first trial, the subject had no logical basis for a 
response, the S and R terms being arbitrarily paired, and 
“meaningless” in themselves. The first trial thus serves 
as an “exposure” trial, or a special form of prompting 
trial to start the learning. On succeeding trials, many 
overt errors were made due to the length and difficulty 
of the task and the absence of “stimulus supports.” Learn- 
ing over a number of trials thus rested on frequency of 
exposure, and number of times correct responses were 
reinforced or “confirmed” and incorrect responses “cor- 
rected” by feedback. As some tasks required stimulus 
learning, response learning, and associative learning, the 
task was hard to master under the condition of confirma- 
tion. This procedure often has been called the “anticipa- 
tion” procedure, as the subject attempts to give the cor- 
tect response before the device presents the confirmation 
(or correction) term. 

2. The Prompting Method. By changing the sequence 
of events in a paired-associate task from stimulus-response- 
confirmation to stimulus-prompt-response, Cook (1956; 
1958, 1960) has found that the prompting method is 
superior to the confirmation method. After exposure of 
the stimulus term, the “response term” is exposed; then 
the subject views the response term and copies it. The 
Superiority of the prompting method has been accepted 
by many only with difficulty, due to the knowledge of the 
traditional confirmation method, which interprets learn- 
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ing as making of responses without direct aid on each 
item as it is exposed, and having them reinforced. The 
prompting method appears to many a less “active” 
method of learning, and thus goes contrary to the belief 
that learning must be an effortful and unpleasant ex- 
perience in which the subject is pitted against the experi- 
menter, who has arranged difficult conditions for the 
subject to master. In a recent study, it further has been 
found that prompting in only three trials out of four 
is superior to either prompting or confirmation alone on 
every trial (Angell and Lumsdaine, 1960). Thus, prompt- 
ing for each “item” separately over a number of trials 
is superior to the more gross prompt represented by Trial 
l in a confirmation condition, and the haphazard prompt 
represented by the presentation of the response term 
when the subject has failed to “anticipate.” And, as 
Angell and Lumsdaine show, performing without the 
prompt on every fourth trial represents sufficient prompt- 
ing, but not over-prompting. This indication that too 
much prompting can be given (the task can be made too 
casy) is important. It may be noted that, in continuous 
discourse material, prompting can be reduced by not 
having a prompt term on some “review” items, or by 


‘using an indirect cue (weaker prompt) in place of the 


exact response term as a prompt. R 
3. The Cue and Context Method. In Skinner-type 


programs, responses are often prompted, as in. Cook’s 
experiments, by presenting the response word before 
asking the subject to write it. Also, less direct forms of 
prompt words are used. These the present writers call 
“cues” (Briggs, 1960). Cues may be synonyms of the 
response term, indirect hints, or other semantic helps of 
various forms. In continuous discourse programs, in addi- 
tion to prompt words and cue words, there also is the 
total context of prose of which the cues and prompts are 
parts. Thus, some words in the frame ask the question 
to be answered, some give background information, and 
some are the “stimulus supports” to help, the subject 
construct the response. The total sequence of material 
is written to change the stimulus context of the responses 
in order to promote understanding or generalization. 
The cue and context aspects of programming of course 
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are generally employed to achieve conceptual learning 
rather than rote memorization. Thus, the objective of 
the program differs from the usual paired-associate con- 
dition. The point is that “programmed” material exer- 
cises more “control” than a textbook because the cues 
and prompts are carefully arranged and overt responding 
is required. This probably serves to focus effort and atten- 
tion upon each “frame” of the program, and prevents 
inattentive reading. 

4. ‘ihe Shaping Method. Skinner has employed the 
concept of shaping in discussion of both human and 
animal learning. In the case of an animal, a reinforce- 
ment is given for any overt response which is “in the 
tight direction,” or is only partially a step toward the 
required response. Reinforcement continues as the animal 
makes increasingly adequate responses. in contrast to 
Skinner’s verbal programming methods, stimulus supports 
are not normally employed for animals. The first behavior 
of the animal in the experimental box is thus considered 
emitted behavior, in the sense that no signal is given 
to stimulate the animal to move about or to seck food, 
etc. Thus, the experimenter puts the animal in a special 
environment and waits for the animal to do something 
before the training (reinforcement) begins. 

In verbal programs, there is not this same deliberate 
effort to reinforce only “partly right” responses. Direct 
signals to perform are given, and stimulus supports (cues 
and prompts) are employed to guide the behavior so 
that each response will be right, even if it represents 
learning of only a small part of a total lesson. Whether 
the term shaping should be employed in both situations 
appears somewhat doubtful. 

5. Reading and Selective Test and Review. Normal 
study methods often consist of rather aimless reading, 
with only perfunctory effort by the student to select im- 
portant points, to repeat them in review, or to organize 
the material in outlines or notes. Pressey developed his 
testing-teachtig devices to improve study by calling atten- 
tion to important points covered in the reading material 
and by presenting questions to be answered. The simple 
devices used with the questions provided confirmation- 
correction, and encouraged the student to correct errors 
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by repeated trials or by re-reading relevant portions of 
the study material. Thus, the total reading material 
formed the context and presented the background 
stimuli. The test stimuli required the student to recall 
and select for himself the information from the reading 
which is most relevant to the answering of the question. 
Questions were not usually interspersed into the reading 
material as in Skinner’s programs, thus placing more 
dependence on the student’s study efforts for effective 
use of the tests. These methods may be characterized 
as less controlled, as the reading units were large and the 
reading more or less separated from the question answer- 
ing. In a sense, however, the method required more 
“construction” or “structuring” by the learner than do 
Skinner’s programs, even though a multiple-choice test 
format was employed. 


Response Frequency: If one of the values of response 
making is to elicit active learning, it is reasonable to 
ask how much response making is necessary. Making 
overt responses may be considered as simply provid- 
ing a general set to participate actively in the learn- 
ing situation. From this point of view, perhaps only 
an occasional response is sufficient to maintain inter- 
est. It is not suggested that an experiment be conducted 
to explore the range of response frequencies in order to 
determine some optimum rate of responding. It is likely 
that the subject matter would influence whatever rate 
would be most effective. Each sub-topic introduced might 
be the best occasion for a response with some material, 
while other units might be most appropriate for different 
types of material. It is worth noting parenthetically that 
a response for each sentence would have the effect dis- 
cussed previously of causing preparation of material to 
focus on student reaction, rather than on the text writer’s 
mode of thinking. 

It is possible, of course, that too many responses can 
be required. Aside from the aversions that can result from 
too much of a good thing, increasing response frequency 
to a point at which little or no benefit is gained reduces 
learning efficiency due to the time required for response 
making. If feedback is provided after each response, the 
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problem of increased time is intensified. If feedback is 
provided after a group of responses, there arise the possible 
problems associated with delayed reinforcement for early 
responses, and the effect of this delay upon correctness of 
later responses. 

For students who already are inclined toward reading 
effectively and with “good concentration,” the introduc- 
tion of response making may constitute more of a disrup- 
tion of the learning process than an inducement for active 
learning. A trade-off in effects may be involved between 
frequent response making in programs versus continuity in 
thought. Skinner has adopted the technique of showing 
“panels” which contain outlines, basic formulae or defini- 
tions, or other materials of a general nature. These 
“panels” are present to view during the regular sequence 
of program events and may mitigate the possible discon- 
tinuity caused by response making as well as afford aid 
to the student in making correct responses. They also 
lend “structure” and may thus enhance understanding 
and rate of progress through the program. 


Response Relevance: Frequency of response making 
raises the consideration of response relevancy. The re- 
sponse may be an important term which is critical to the 
learning outcome or a term which has little or no bearing 
on what is to be learned. While it would seem apparent 
that choice of the response terms to be used is a most 
critical aspect of programming, the value of response 
making per se can be evaluated by requiring such per- 
functory responses as articles, conjunctions, and other 
routine parts of sentences. 

Responding with perfunctory or non-crucial words cer- 
tainly could tend to increase the rate of progressing 
through a program, and may lend readiness to read effec- 
tively a key word to follow. In general, however, it is 
likely that the subject matter or desired learning outcome 
determines the importance of response relevancy. The 
more the response term is identified with or synonymous 
with the desired learning outcome, the more important 
should become the requirement of a relevant response. 
Even if the response term does not constitute that which 
is to be learned, highly relevant responses certainly aid 
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the student in deciding what aspects of the learning ma- 
terial represent the core of what is to be learned. Even 
a signal to underline key words may have a similar effect. 


Repetition: It is important to keep rate of responding 
clearly distinguished from response repetition. If the 
response term is closely identified with the desired learn- 
ing outcome, such as in spelling technical terms, it is 
apparent that much advantage can be gained from requir- 
ing frequent repetition of the response. Requiring the 
same response to be made a number of times increases 
response strength with number of repetitions. A program 
sequence provides a convenient format for utilizing this 
principle of learning, and may avoid boredom as compared 
to repetitions in the identical context. Nevertheless, the 
nature of the criterion situation may determine whether 
repetition in the same or varied contexts will be most 
effective. Since this can be a very powerful variable, it is 
most important that, in establishing the importance of 
other variables, such as “size of step,” there is no experi- 
mental confounding involving the response repetition 
variable. Also, it may be assumed that changing the 
context for the repeated response helps achieve transfer. 


Size of Response Segment: Size of response is another 
variable which is intimately tied to the desired learning 
outcome. Spelling, for example, leads us to consider 
smaller response units than would English compositions. 
Nevertheless, in our approach to any subject matter, we 
have the option of choosing the size of our response unit 
and the possibility of some systematic change in size of 
response during the learning procedure. We will not con- 
sider here the issue of part-whole learning. The size of 
response, however, is relevant to the occurrence and im- 
plications of making errors and to the scope of elicited 
response pattern. 

Much has been said of the value of “constructing” 
a response, but this value has not been calibrated experi- 
mentally in terms of true individual-imposed organization 
of materials, brief effortful fill-ins, and perfunctory copy- 


ing responses. 
Many real-life performance requirements call for large 
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units of response, as in making a speech, building a house, 
painting a picture, etc. Perhaps size of response should 
be increased as subjects gain knowledge and proficiency 
in an area of study and performance. One learns to spell 
words one at a time, perhaps a letter at a time. Speed 
in handwriting later is developed, until spelling of words 
often drops out as conscious effort. Whether different 
learning conditions are fequired for these different skill 
Jevels has not yet been fully established. 


Location in the Frame: The response may be required 
at various positions in the frame—from the first word 
to the last word in the frame. This position is determined 
generally by the syntax or logic of the material. Position, 
may have other special relevance. 

A response near the end of a frame benefits from all 
cueing or prompting in the frame when read in the 
normal fashion. All material in the frame is read with 
the motivation of having to respond correctly. The “re- 
inforcement” follows the response with little delay. 

A response near the beginning of a frame may be made 
without use of the response aids which follow it. Or the 
following material may be read until the response is 
brought to threshold. That is, the response requirement 
may establish a specific set to acquire the information 
needed to make the response. The response can be made 
as soon as it becomes sufficiently cued for that student 
(a device for supply close-to-threshold response aid). 
‘This may result in a unique read-study sequence through 
the frame material for a student. If material following 
the response is read or re-read after the response is made, 
it would result in a delay in the reinforcement. 

If more than one response is required in a frame, the 
relationships of response position and of response func- 
tion become more involved. One interesting possibility 
is that multiple blanks in a frame may lead to overt 
Tesponses in opening portions which become more power- 
ful cues to the final response than these same words 
written into the frame. Thus, a successful constructed 
Tesponse may be a more direct prompt than supplying 
the word in the program. But, if the response asked for 
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early in the frame cannot be made correctly, it may be 
better that the word were supplied in the program. 

More generally, it is possible that an initial fast reading 
of the program preceding a second trial to fill in the 
blanks would employ the whole first (reading) trial as a 
cue for the correct responses on trial 2. If this were found 
successful, reduced cueing may be usable. Another pos- 
sibility is that the subject should write the response if 
he believes he knows the answer, but “peek” at the feed- 
back term if he is uncertain. 


Remoteness of Prompts: A study now is being planned 
to determine whether moving prompts and cues to an 
early frame, but requiring a response in a later frame, 
is more effective than having the prompt present to view 
at the time the response is required. 

Having responses depend on learning material that is 
not available to the student at the time the response is 
required should motivate him to read material carefully 
and thoroughly with more than the goal only to make a 
particular correct response. This advantage is in contrast 


to shaping a particular response. 


Response and Feedback Relationships: Different com- 
binations of response conditions and feedback conditions 
may shift the emphasis from whether it is the reading of 
stimulus (leaming) material, the making .of explicit 
responses, the making of implicit responses, or the receiv- 
ing of feeding-back that is most critical or important in 
producing the desired learning. A basic point here is 
that, in evaluating the effects of a response variable or a 
feedback variable, it is necessary to identify what other 
program characteristics seems to have been associated 
with this variable by programmers, and to determine 
whether association is invariant. If the associated program 
characteristics are found to be tied necessarily with the 
variable under consideration, that variable mast be related 
to both performance criteria and format criteria—with 
the latter involving variables which can, in tum, be 
related to performance criteria. 
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Form of Response: It is quite likely, however, that we 
will find that, in many cases, there is no basis for believ- 
ing that a fixed combination of variables is universally 
optimum. Consider as an example the form of response 
required. The response may assume a wide variety of 
forms. It may be written, spoken, throught, constructed, 
selected, etc. There has been much controversy over 
whether a written constructed response is preferable to a 
response selected from a set of multiple-choice alternatives. 
These two forms of responding have been associated with 
specific characteristics of learning material presentation 
and with specific kinds of feedback that are possible. It is 
feasible, however, to consider the multiple-choice alter- 
natives as a form of cueing or prompting. Many students 
may, in fact, develop their own set of response alterna- 
tives from which to select when none are formally pro- 
vided as such. Conversely, when the multiple-choice al- 
ternatives are provided, the student may construct a 
response before proceeding to “select” one. Also, the 
selection could be indicated by writing out one of the 
alternatives instead of by making some more simple motor 
response. To break down the apparent fixed differences 
implied by these two forms of responding even further, 
we might introduce the familiar “none of the above” as 
an alternative. 

It is contended that form of response can be studied 
as a variable under conditions which free it from many 
of the closely related issues of programming and which 
will allow a more useful evaluation of the criterion situa- 
tions for which various response forms are useful. Whereas 
large steps and multiple-choice format have been asso- 
ciated historically in Pressey’s procedures, while small 
steps and constructed response characterize Skinner’s 
methods, there is no obvious reason why these particular 
combinations should be maintained. 


Response Difficulty: The ease with which a student is 
able to make the correct response is a point deserving of 
special attention. Response difficulty is another variable 
which has engendered much controversy. One view is that 
a correct response cannot be made too easy for the sub- 
ject. A second view is that the item should be difficult, 


Ll 


R. A. Gorpseck anp L. J. Brices 183 


but not so hard it will be failed. A third view sees a trade- 
off between the positive value of effort exerted on a 
difficult item and the negative effect of an error. A fourth 
view is that the experience of making errors and being 
required to correct them is an essential practical aspect 
of the learning process. For the sake of making program 
writing and program revision a simpler task, it would be 
fortunate to find either that making correct responses 
should be extremely difficult for the student, or that the 
student should have no difficulty at all in making a 
correct response. There is some evidence, however, that 
the optimum point of difficulty lies somewhere between 
these extremes (Goldbeck, 1960). A reasonable place to 
expect this point to lie is near the threshold for the 
response. But, how close to the threshold? Will it be 
found necessary to develop techniques which can bring 
different students to different precise levels for responding 
correctly? The answers to these questions depend very 
much on whether we find that the response made should 
always be the correct one, or that the occurrence of 
wrong responses is acceptable and perhaps under some 
conditions even beneficial. aut 

The danger of heavy cueing to give a high likelihood 
of correct response is that response making may become 
an automatic activity with none of the advantages of 
active participation. ‘ aoe 

The danger of less cueing to give a lower likelihood 
of correct response is that frequent errors might also 
discourage active participation and that errors might 
lead to misconceptions. Corrective feedback is one im- 
portant safeguard against the latter danger. An argument 
which favors a moderate degree of response difficulty is 
that the implicit response pattern induced by a challeng- 
ing response requirement enhances the effectiveness of 
feedback information. Present methods of revising pro- 
grams by tryout succeed in correcting items too difficult, 
but fail to eliminate items which are too easy. Perhaps 
a systematically programmed combinatio of response 
difficulties above and below threshold can provide the 
advantages put forward for each point of view. 


Functions of Feedback: Much of the preceding dis- 
cussion has touched in general on the role of feedback 
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in programmed instruction. Specifically, feedback may 
serve several functions: 
1. It may provide information concerning the adequacy 
of responses made. 
2. It may serve as a reinforcement and reward for 
responses. 
3. It may have a motivating effect on performance. 
4. It may be used to direct the next step to be taken 
in the learning program. 


In regard to the information characteristics of feed- 
back, a correct response may be confirmed and an incor- 
rect response simply may be disconfirmed, or it may be 
corrected. The confirmation of a correct response can be 
indicated by a simple signal such as a light or the word 
“correct” which can be used for all correct responses. 
The confirmation may contain somewhat more informa- 
tion if the correct answer is used as the feedback. This 
usage provides additional exposure to the correct response 
and, when used with written responses, can provide cor- 
rection of minor errors such as spelling mistakes. 

Crowder has adopted the practice of including an 
explanation of why the response is correct, thereby taking 
advantage of this part of the learning process to introduce 
additional leaming material or to insure that the correct 
response is fully understood. There is another way in 
which some statement concerning the correct response 
could be used to provide feedback. The statement might 
not include the correct response or a direct confirmation 
of it, but could be written in such a way that the student 
would be encouraged to reconsider the stimulus or learn- 
ing material and his response to determine that he has 
responded correctly. Such a statement may take the form 
of “If your answer means that ‘such and such’ is true, 
then you are correct.” The advantage of this type of 
technique is to promote a more active and comprehensive 
pattern of student participation. The technique also 
should help gliminate the problem that arises when a 
variety of synonyms constitute a correct response or when 
a large-size response (e.g., a sentence) is required. There 
may be some parallel between this “indirect” form of 
feedback and the practice of intermittent reinforcement 
and use of subtle cues. 
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The simple disconfirmation of an incorrect answer is 
accomplished by some signal which can be used for all 
incorrect responses. The possible advantages of the dis- 
confirmation technique are that the student is discour- 
aged from using the feedback as a substitute for a serious 
attempt to make a correct response, and that it allows 
for the possibility of requiring additional attempts to 
respond correctly without the uncontrolled direct prompt 
given by the feedback. 

The correction of an incorrect response can be ac- 
complished in much the same manner as a confirmation: 
present the correct response, explain why the response 
is wrong, or present a statement that would not be con- 
gruent with the incorrect response. As discussed earlier, 
if the program is one which is prone to result in a sub- 
stantial number of incorrect responses, the correction 
feature of feedback becomes a key point in the learning 


process. o 
The reinforcement or reward characteristics of feedback 


have been dealt with extensively in the laboratory situa- 
tion and are basic to Skinner’s development of the shap- 
ing concept for programmed classroom instruction. The 
establishment of desired stimulus-response connections 
by a process of systematic reinforcement of intermediate 
stimulus-response connections raises many of the con- 
siderations found to be relevant in the laboratory. The 
schedule of reinforcement, the specificity of reinforce- 
ment, and the extrinsic versus intrinsic properties of 
reinforcement are among the considerations for study 
in the programmed instruction format. i 

The motivating properties of feedback constitute one 
of the most powerful ingredients for success with pro- 
grammed instruction. Finding that they are able to re- 
spond correctly consistently may be a strong motivating 
factor for many students, especially those who, in the 
past, have found learning to be a very frustrating experi- 
ence. For students who have experienced much success 
in learning and who may have found that learning situa- 
tions have not enabled them to utilize fully their capa- 
bility for success, conditions which provide consistent 
Teinforcement may tend to have a negative motivating 
effect. Regardless of such individual differences, there 
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remains the question of for how long a period positive 
reinforcement will retain a motivating effect. Studies 
aimed at enhancing the motivating effects of feedback, 
especially on a longterm basis, are needed. Perhaps some 
reduction in feedback as learning progresses is consistent 
with another prevalent concept, namely that less guid- 
ance should be given as learning progresses. 

The feedback term may have the function of directing 
or guiding the learning process in either an explicit or 
informal manner. The feedback can include the instruc- 
tion to proceed to some later frame in the sequence if 
the correct response has been given, and it can indicate 
mastery of material in the intervening frames. If an in- 
correct response has been given, the feedback can include 
the instruction to return to some previous frame or to a 
special remedial branch of the program. 

Even without the formal branching usage, the student 
can make use of the feedback to recognize the need to 
review previous frames or to adjust his rate and effort in 
proceeding through the program. Perhaps some of the 
complexities in branching format can be avoided by 
developing and providing the student with simple criteria 
for altering his progress through the program, in order 


that he may make most effective use of the programmed 
material, 


POINTS OF EMPHASIS FROM PROJECT 
EXPERIMENTATION 


Results from the first experiment completed under 
this contract provide a basis for additional emphasis here 
upon some of the points which have been made earlier 
in this report. Rather than summarizing all points made 


gaien the following items are selected for special com- 
ment. 


The experiment, teported earlier by Goldbeck (1960), 


found that, for a program of discrete, factual items, 
the effects of’ overt written Tesponses, covert responses, 
and reading responses vary in accordance with two criteria 
employed and in accordance with item difficulty. Three 


levels of difficulty were achieved P 
cueing and prompting, ieved by various degrees of 
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When retention test scores are plotted without regard 
for time taken on the learning trial, the overt response 
group performed poorer than the others for easy items, 
but much better than the others on items of medium - 
difficulty. But, the overt group required more time for 
the learning trial, as may be expected when written 
responses are required. 

When retention test scores are expressed as number 
of test items correct per minute of learning trial time, 
the reading group is superior for all difficulty levels of 
items. From the findings of this experiment, several 
comments may be made in the context of issues discussed 
in the present paper. 

First, we observe that the writing of a correct con- 
structed response in a program appears to demonstrate 
that some learning already has taken place, providing 
the response is not simply a copying response. The ex- 
planation for this apparent learning must lie in implicit 
activity. The writing of the response may or may not add 
strength to the retention of that which is learned, depend- 
ing apparently on the extent of the cueing or prompting 
(Goldbeck, 1960). l 

The interaction found in the experiment between dif- 
ficulty level and mode of response, in terms of a reten- 
tion criterion, appears to be related to concepts such as 
effort, attention, speed of reading, and distribution of 
study time spent in reading, thinking, answer forming, 
answer recording, etc. The experiment failed to find 
overt responding with feedback a necessary condition for 
effective learning, at least for some materials and condi- 
tions. Similarly, we find that the making of an error 
during the learning trial does not always signify that the 
same (or any) error will be made in the subsequent 
retention test. : 

Apparently, the negative effect of some overt eect 
overcome by the positive effect of some portion o! e 
implicit activity preceding the recording of a one 
answer and by the effect of the feedback term. ic 
hypothesized that, when an item is moderately difficult, 
and effort is exerted to give the right response, a ary 
of implicit responding results, some in tight lines o 
thinking and some in “wrong.” If wrong predominates 
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in strength or if strength oscillates, a wrong response may 
be recorded, but the resulting feedback is employed by 
the subject to confirm the correctness of the right implicit 
tendencies. In other cases of wrong overt responses on the 
learning trial, an error may be made on the retention 
test. A breakdown of right and wrong responses on the 
learning trial, for three difficulty levels, in relation to 
correctness on the retention test, has been presented 
previously (Goldbeck, 1960). 


IMPLICATIONS FOR EDUCATIONAL PRACTICE 


Methods of Programming: From a consideration of 
many of the programmed instruction variables especially 
bearing on response making and feedback, it is indicated 
that we are not ready to be content with any one “pack- 
age” of instructional conditions, however much suc- 
cess and enthusiasm it may have enjoyed. We yet have 
to establish fully the relative importance of variables, the 
variable characteristics which contribute most successfully 
to desired learning outcomes, and some of the combined 
effects of these variables. It is also reasonable to expect 
that new variables will be identified and that more useful 
definitions of current variables will be developed. As 
our knowledge about the ingredients of successful pro- 
grammed instruction techniques grows, we can apply this 
knowledge to the improvement of existing programs and 
to the development of new program techniques. 


Combining of Molar and Molecular Approaches: While 
we may view the advancement of programmed instruction 
techniques as depending heavily on the investigation of 
the relationships between specific learning variables and 
desired learning outcomes, it is necessary to recognize 
that the course of experimentation will be affected by 
theoretical predilections and frames of reference. The 
learning experimentalists differ in their views on learning 
just as educators’ beliefs about how children learn vary 
considerably. Some “good” teachers are said to be 
successful because they “challenge” students. Others 
are demanding in a way which provokes high student 
effort. Some lead the student through the problem. Others 
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hold individual and group drill until mastery is achieved. 
Many use discussions to promote transfer. , 

One may see the “molecular” approach of emphasizing 
simple habit formation and the “molar” approach of em- 
phasizing broader and more generally defined units of 
behavior in both the laboratory and classroom settings. 
Whereas the leaders in education have tended to ex- 
hibit concern with the more broadly defined aspects of 
learning, the contribution to education by programmed 
instruction techniques has been based largely on the 
laboratory work with relatively simple learning habits. 
Hopefully, both educators and experimentalists, each 
with points of view varying in molarity of approach, can 
teconcile their conceptual differences under the impetus 
to increase their understanding of the new instructional 
technique and to improve its effectiveness to deal with a 
broad range of subject matter. 


Programming and Creativity: It is pertinent to point 
out that using programmed instruction techniques for 
effective learning of basic curriculum materials is not an 
assurance that these same techniques are appropriate for 
more highly “intellectual” or creative behavior. It is pos- 
sible that different forms of programmed techniques must 
be developed for the different levels of education. It is 
also possible, however, that the learning techniques can 
be quite similar and that only differences in the program- 
ming of the subject matter will be needed. In either case, 
we must consider the effects of programmed instruction 
variables on how the student learns “how to learn.” It 
should be noted parenthetically that such considerations 
also may be relevant to effective learning of basic curricu- 
lum material. 

Although research is in progress to use programs io 
stimulate “creativity,” we yet do not fully understan 
the processes and the variables which produce creative 
work. Some creative people have had no formal training, 
and haye been reared in an intellectually “impoverished 
home. Although some information must be employed in 
creative work, overtraining, it is suspected, may inhibit 
creativity if overtraining contributes to rigidity. Yet, Pe 
fessional people with much education often out-crea 
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untutored persons or those with “technician” training. 
The subject matter of education in the professions differs 
from that taught to the technician, with, we suspect, 
superior results. Thus, “education” is broadening in terms 
of transfer while “skill” training channelizes or controls 
behavior. 


Some Unanswered Questions of Broad Significance: 
What are the implications of these considerations for 
“controlled” programs versus “stimulating” programs? 
Would “controlled” programs produce a generation of 
uniformity and inhibit activities which have not been 
programmed, but which may be needed to meet future 
problems? Would too much emphasis upon simple habit 
formation as a means of education inhibit new frontiers 
of thought? These questions will be asked by educators 
and the public. 

Individual differences are known to exist, but evidence 
is conflicting as to whether differences are increased or 
decreased by training. If these differences are needed for 
the good of society, is there any possibility that there 
could be too much use of controlled programming? Or, 
to the contrary, would basic sçience be better mastered as 
a start toward new developments under more highly 
controlled education? 

It is the writers’ contention that obtaining answers to 
such general questions need not be restricted to the 
philosophical or theoretical level of inquiry. Experimental 
analysis of the variables involved in programmed instruc- 
tion, of the sort discussed in this paper, will contribute 
to the increased effectiveness of programmed instruction 
techniques. This effectiveness, in turn, must be evaluated 
in terms of educational goals. Thus, a balance between 
empirical tests of programming methods and research to 
improve programming technique should pay off in the 
long run. 

Lest the reader feel that some of the questions raised 
above contain too much of a “viewing with alarm” of 
the unknown longterm effects of controlled programming, 
we hasten to state that we advocate sufficient experience 
with the methods to help answer our own questions. Also, 
there will be certain checks and balances which accom- 
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pany trial of programs. Normal procedures will not be 
abandoned overnight. Whether due to cultural lag or to 
wisdom, the first widespread use of programming in 
education will be as another tool to help get the educa- 
tional job done. Thus, a “systems” approach, whereby 
programming is incorporated into total procedures where 
it proves most useful as a component of total methods, 
will result in a favorable balance of procedures. Even- 
tually, educational objectives must be studied systemati- 
cally in terms of methods effective in achieving them. 
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A Comparison of Several Types of 


“Immediate Reinforcement” * 


J. Wurm Moore anp Wenpbe tt I. SMTH 
Bucknell University 


An application of reinforcement theory has received 
particular attention from those working in the field of 
automated instruction (Coulson and Silberman, 1960; 
Lawson, 1960; Lumsdaine, 1960). Specifically, much of 
the demonstrated effectiveness of auto-instructional ma- 
terials has been attributed to immediate knowledge of 
results (KR) as a reinforcer. 

While the value of immediate reinforcement in in- 
creasing the rate of learning has been demonstrated, the 
assumption that immediate reinforcement (when the 
reinforcement consists of KR) as defined in the animal 
laboratory or in motor learning tasks is synonymous 
with immediate reinforcement through KR as used in a 
verbal learning task, may be questioned (Moore & Smith, 
1961). 

Most of those who have questioned the reinforcing 
properties of KR, have not questioned the general con- 
cept of reinforcement, but they propose that a distinction 
be made between confirmation and reinforcement of the 
response. Carr (1959) states that “confirming the cor- 


* This preliminary report is based on a study supported by 
the Air Force Office of Scientific Research, Grant No. AF- 
AFOSR-61-54, and is used here with the permission of the 
authors. 
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rectness of a learner’s responses to problems may be 
expected to be reinforcing only if the learner’s motiva- 
tion is intrinsic to the task being learned.” 

This problem is further complicated (whether KR is 
classified as confirming or reinforcing) by the limited 
amount of empirical evidence available concerning the 
effect various methods of providing KR have on human 
learning. For example, KR can be provided by (a) the 
immediate knowledge that the given response is correct 
(a flashing light for a correct response), (b) immediate 
knowledge of what the correct response should be (the 
correct answer to the stimulus item appearing immedi- 
ately after the student responds to the item), (c) im- 
mediate reinforcement for correct responses using an 
extrinsic reward (a piece of candy or a coin presented 
for each correct response), and (d) KR provided by cues 
which appear in succeeding items (a normal condition in 
the linear program utilizing the concept of successive 
approximations for increasing the likelihood of making 
the correct response). In the last example, this method 
of providing KR could be in addition to, or in lieu of, 
the conventional method of providing KR in programmed 
instruction. 

From the examples cited, it can be seen that methods 
for providing KR vary greatly in their reinforcing and 
in what may be called their “guidance” properties. For 
example, one would expect a tangible reward (money) 
to have greater drive reducing properties for some people 
than would the more intangible reward of KR. 2 

Further, if KR, where the correct response appears 1m- 
mediately after the student makes his response, has only 
“guidance” properties, then this type of KR is unneces- 
sary (assuming that the linear form of programming Is 
being used). This is based on the assumption that suffi- 
cient cues should be built into the successive approxima- 
tions of the items to assure a high probability of making 
the correct response. 

The purpose of this investigation was to compare the 
effects of various types of KR in leaming psychology 
from programmed material. 
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PROCEDURE? 


Subjects: Two sections totaling 220 university students 
enrolled in introductory psychology for the second term 
of the 1960-1961 school year were randomly assigned to 
one of 10 groups. To reduce the effects of the teacher 
variable (there was a different instructor for each section) 
equal numbers of Ss within each section were assigned to 
each group. The Ss remained in their respective groups 
throughout the experimental period. To. insure com- 
parable motivation for all groups, Ss were told that a 
part of their semester grade would be based on their 
eee estat on examinations based on programmed ma- 
terials. 


Experimental Period: The experimental period con- 
sisted of six weeks beginning with the second week of 
March 1961. The students completed programmed mate- 
tials on a regularly scheduled basis, one hour on two 
days a week during the six-week period. In no case were 
the laboratories? scheduled on two successive days. All 
students were expected to complete 96 frames during 
each laboratory. 


Experimental Groups: Since the amount of learning as 
measured by two tests of achievement could be related 
not only to the types of KR, but to the type of response 
required (i.e. whether multiple-choice or constructed), 
experimental groups were established to compare the 
effects of various types of KR on both multiple-choice 
and constructed-response items. Further, two more groups 
were established to compare the effects of teaching 
machines on learning.’ In total, 10 experimental groups 


1 Our appreciation is expressed to Douglas Candland and 
Mary Jane Mordan for their extensive participation in the 
conduct of thi> experiment. 3 ; 

* Laboratories refer to those periods during which students 
worked on programmed material. S 

3 Komonski and Eigen (1960) completed a study comparing 
programmed texts and teaching machines and did not obtain 
significant differences on the criterion measure. 


J. W. Moore anp W. I. Smau 195 


, were established. Experimental groups one (E,mc), two 
(Eəmc), three (E3mc), four (E,mc) and five (E;mc) 
used programmed materials with multiple-choice (mc) 
type answers. In group E,mc-E,mc the material was 
presented in a teaching machine, varying only the method 
of providing KR. In the fifth group the material was 
presented in a programmed text form with KR provided 
in the form of the correct response. Experimental groups 
one (E,cr), two (Eyer), three (Eger), four (Eyer) and 
five (Escr) used programmed materials which required 
constructed-responses (cr). As with the multiple-choice 
groups, teaching machines were used with the first four 
constructed-response groups and a programmed text was 
used for the fifth group. The 10-celled chart shown here 
presents the experimental groups with the various condi- 


tions and types of KR. 


EXPERIMENTAL DESIGN 


Con- 


Knowledge of Results | Multiple | structed 
Choice | Response 


No knowledge of re- Ei Ei Only stimulus mate- 
sults rial 
Immediate knowledge Es E: The correct answer 


appearing after 
the response had 
been made 


of what the correct 
response should be 


Maadi SAAR 

Immediate knowledge 
that the given re- 
sponse is correct 


Es Es Stimulus materials 
plus a flashing 
light for correct 
responses 


E: E: One penny for each 
correct response 


Immediate knowledge 
of results plus ex- 
trinsic reward 


Immediate knowledge Es Es Programmed text 


of what the correct 
response should be 


eee ee |e | err 
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Facilities and Equipment: Two regular classrooms were 
scheduled for use by the 10 groups. Foringer write-in 
machines were used for presenting the materials to the 
constructed-response groups. The E,cr group machines 
had black masking tape covering the space on the ma- 
chine where the correct response normally appeared. This 
prevented S from knowing whether the response was cor- 
rect or not, Escr’s used the Foringer machine in the con- 
ventional fashion (the correct answer appeared immedi- 
ately after S made his response and pulled a lever). The 
E,cr group used a machine equipped exactly like the 
Eyer group except that a small light was attached to the 
upper right-hand corner of the machine. This light was 
controlled by an O seated behind S and for each correct 
response which S made O flashed the light, signaling that 
the response was correct. If the light was not flashed 
immediately after S made his response, S knew that he 
was to move to the next frame. In no case did O com- 
municate orally with S. The E,cr machine-observers com- 
bination was exactly the same as the E,cr; however, O 
kept a running tally of the correct responses. At the end 
of each laboratory period S was paid (by the laboratory 
proctor) one penny for each correct response. The E,cr 
group did not use the Foringer machine but used a pro- 
grammed textbook. 

The multiple-choice groups used specially designed 
three-choice-response teaching machines. The programs 
were placed in booklet form to the left of the machine. 
On the face of the machine (between two rows of re- 
sponse buttons) an answer sheet containing three possi- 
ble responses for each frame was placed. The Ss were 
told to select a correct response, circle it on the answer 
sheet, then press the corresponding response button on 
the machine. The Eme group used a machine which 
kept a record of the number of responses but did not 
provide KR. Egmc used a machine equipped with three 
lights (keyed to the three choices on the answer sheet). 
The machine was so constructed that regardless of 
whether the correct response button was pressed, the light 
indicating the correct response would light. The machine 
also kept count of the number of responses which S 
made. Egmc used a machine which differed from the 
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Escr machine in that it had only one light on the panel. 
When the correct response was made the light would 
light; when incorrect, it did not light. This machine was 
also equipped with counters which not only counted the 
total number of responses the Ss made, but also kept 
count of the total number of correct responses. The 
E,mce group used a machine exactly like the Emc and 
was paid one penny for each correct response. This pay- 
ment was made at the end of each laboratory period by 
the proctor. The Esme group used the same materials 
and followed the same procedures as the other multiple- 
choice groups. A record was made of the numbers of cor- 
rect responses and total responses for each S at the end of 
each period. 

Each laboratory had one proctor assigned for its gen- 
eral supervision (the distribution of materials and direct- 
ing of Os). An average of one O for each three Ss was 
provided in each of the laboratories. The purpose of these 
Os was to make certain that the Ss completed the pro- 
gram as stated in the experimental design. The presence 
of several Os helped to reduce the “observer variable” 
introduced in the Eyer and E,cr groups.* In no instance 


Materials: Eleven hundred fifty-two frames of the 
Holland-Skinner psychology program were used.’ This 
program was selected because of the careful manner in 
which it was developed. The constructed-response_por- 
tion consisted of placing 96 frames on each of 12 rolls of 
materials for use in the Foringer machine. One roll was 
used during each laboratory session. Since the conven- 
tional Holland-Skinner program is in the constructed- 
response form, a modification of the program had to be 
made to adapt it to the multipie-choice machines. This 
modification consisted of constructing three answers for 
each item, only one of which was correct. Otherwise, the 
programs were exactly alike. , 

No other formal instruction in learning theory was 
provided for the Ss during the experimental period. 


‘In these groups one O was necessary for each S. 
did Os provide information about the program to me 
? Permission to use this material was granted by James ol- 
land to whom we express our appreciation for this courtesy. 
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Evaluation: All Ss were given an examination at the 
end of the sixth unit of material. This examination was 
administered in one of their regularly scheduled class 
periods. The examination was based solely on the infor- 
mation included in the six units and the test items were 
closely related to the items contained in the program. 

At the end of the experimental period, a final examina- 
tion was administered which included all of the concepts 
presented during the experimental period. It was made’ 
up of one-half multiple-choice items and one-half con- 
structed-response items. This type of examination, in ad- 
dition to being a measure of achievement could provide 
some information concerning the interaction between 
the type of response element employed in the program 
and the type of test item used in evaluating the student. 

An attitude measure, in the form of a questionnaire, 
was completed by each S at the end of the experimental 
period. It was expected that this might indicate some 
relationship between the achievement, as measured by 
the criteria measures, and the interest in this method of 
instruction. 

Results: Since this is a preliminary report, only data 
derived from the two criteria measures are available at 
this time. An analysis of variance was applied to the data 
to compare differences of means resulting from the types 
of KR provided, the method of responding, and the in- 
teractions of these variables. Tables 1 and 2 summarize 
the results of this analysis. Since none of the F’s reached 
the necessary level of significance it must be concluded 
that differences among groups are attributable to chance. 


DISCUSSION 


The failure to attain significant differences may be 
attributable to several factors. First, the evaluating in- 
struments used as the criterion measures may not have 
been sufficiently sensitive to the effects of the various 
experimental “conditions. Second, differences betwen 
and/or among groups may have been reduced by inade- 
quate control of certain variables within the experimental 
design (e.g., the ratio of Os to Ss varied from one per S 
to one for three Ss). Third, the procedure followed by 
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TABLE 1 


Analysis of variance of achievement test scores (test I). 


Source of Variation df ms F 
Mode of response 1 33.49 1.16 
Knowledge of results 4 59.99 2.09 

_ Interaction 4 11.73 0.41 
Within 177 28.75 
TABLE 2 


Analysis of variance of achievement test scores (test IT). 


a Source of Variation df ms Bs 
Mode of responses 1 29.60 0.17 
Knowledge of results 4 372.11 2.12 
Interaction 4 29.45 0.17 
Within 180 175.48 


= some Ss in working with the material may have reduced 
differences. For example, it was possible for those Ss who 
used multiple-choice teaching machines and programmed 
texts to refer to a previously completed frame (one of the 
functions of the Os was to see that this was not done), 
while referral to a previously completed item was not 
possible for Ss who used constructed-tesponse machines 
* (Goldbeck, 1960). 

f the criteria measures were valid, and if the failure 
to reach a satisfactory level of significance was not the 
result of uncontrolled variables, then other explanations 
are possible. For example, the comparisons made be- 
tween groups using the constructed-response machines 
with and without KR, are based on well-controlled experi- 
mental conditions. Two, or the combination of two, hy- 
potheses may account for the failure of these comparisons 
to differ significantly. First, the “no KR’>groups may 
have received KR in the succeeding frames (Moore and 
Smith, 1961). If this is the case, it could be questioned 
x whether this type of KR was “intrinsically reinforcing.” 
(For KR of this type to be intrinsically reinforcing, it 


ay 
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would seem logical to assume that enough information 
would have to be obtained from the frame to provide * 
absolute assurance that the preceding item was correct), 
or whether the KR was sufficiently subtle that it provided 
some guidance in answering the item but still was not 
intrinsically rewarding. Second, the group which was not 
provided with immediate KR may have been forced to 
respond more actively to the individual stimulus elements 
than the group which had immediate KR (Goldbeck, 
1960). If this were the case, it could be assumed that this 
effect was sufficient to counterbalance the positive effects 
of the intrinsically motivating properties (if, in fact, this 
is a result of KR) of immediate KR. A further finding 
resulting from that part of the study in which better con- 
trols were possible, was that the groups using the pro- ^ 
grammcd text and those using teaching machines did not 
differ significantly. In fact, the mean achievement scores 
on the criteria measures were consistently in favor of the 
programmed text. This finding has been reported by 
Komonski and Eigen (1960), also. 

An interesting observation was made concerning the 
rank of the mean number of errors on each of the two 
tests for both the constructed-response and multiple- 
choice groups. It was noted that a smaller mean number 
of errors was made (rank) by the groups using the pro- 
grammed texts and the groups which were receiving an 
extrinsic reward than was made by the other groups. If 
significance can be attached (because of relevant posi- 
tions of mean errors on the criterion measures) to this 
observation, it suggests that (1) the teaching machines 
are not as an effective a media (in terms of learning) for 
presenting programmed materials as is a programmed 
text, and (2) an extrinsic reward (though in this case it 
might be better classified an incentive rather than a 
reinforcer), when added to KR, is more effective in 
oe ae error rate on a criterion measure than KR by 
itself. 

At the time of the writing of this preliminary report, a 
complete analysis of the data has not been made. How- 
ever, an analysis of error rate on programmed material is 
being done, as well as a tabulation of the results from a 
questionnaire which was designed to provide an evalu- 
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ation of interest and attitude toward the experimental 
methods used in this investigation. Comparisons will be 
made of the types of KR and the types of responses on 
the error rate of the programmed material. Comparisons 
also will be made relating error rate on programmed ma- 
terial and scores achieved on the criteria measures. The 
questionnaires will provide some information concerning 
the relationships between interest in the programmed 
material and the level of achievement defined in terms of 
error rate and of scores on the criterion measure. 
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Size of Step and Cueing* 


Wenpett I. Smita ann J. Wittram Moore 
Bucknell University 


One of the common problems of a programmer who 
is using the linear form of programming is to determine 
the size-of-step between frames. The concern about the 
concepts of size-of-step and cueing is based on the desire 
to increase the probability that a student responding to 
any given item will make the correct response. The size- 
_of-step may vary in at least two ways, It may vary in the 
relationship of a succeeding item to a Previous item (the 
degree of change in the reduction of cues in the items), 
and, two, it may vary in the number of cues within an 
item that is not dependent on a previous item. 

The concept of size-of-step, by definition incorporates 
the principle of cueing and it may, in addition, influence 
the rate and amount of learning through the frequency 
with which repetition of the correct response is required. 
On the other hand, the concept of cueing places less 
emphasis on learning through the repetition of the cor- 
rect response in a series of frames, and places more atten- 
tion on the eliciting of the correct response within a 
given item. It is on this basis that the concepts, size-of- 
tep and cueing are differentiated in this paper. 

The importance of this problem is evident when atten- 


* This is a preliminary report of a study supported by 
stant No. 736087, United States Office of Education, Depart- 
nent of Health, Education and Welfare, and is used here 
vith the permission of the authors. (A complete report on this 
esearch will be found in Psychol. Rep., 1962, 10, 287-294.) 
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tion is given to the conventional methods of program- 
ming in the linear form. Usually the size-of-step (i.e., the 
number of frames included in the program) is deter- 
mined by the slowest (lowest ability) test student used 
in the development of the program. The assumption 
usually made is that if the size-of-step is decreased to the 
point where the weakest student can complete the mate- 
rial with a minimal error rate, the optimum program for 
all students for whom the program was written will have 
been developed. 

Some evidence is available (Goldbeck, 1960) which 
indicates that a low error rate on a program does not 
necessarily mean optimum achievement on a criterion 
measure. This suggests that providing larger steps (either 
by a smaller number of repetitions in responses for a 
given concept or by reducing the cues within a frame) 
does not necessarily reduce the efficiency in learning, 
although it may increase error rate. Therefore, it is de- 
sirable that more attention be given to the performance 
on the criterion measures as a means of evaluating pro- 
grammed material than to the error rates for a program. 
This is not to imply that a relationship does not exist 
between the error rate a student makes on a program 
and the score he achieves on a criterion measure, but it 
does suggest that excessively small steps and/or strong 
cueing (to assure minimal error rates) may not produce 
the highest scores on a criterion measure. 

The purpose of this study, therefore, was to compare 
the effects of three different sizes-of-step, amount of cue- 
ing, and the use of illustrations on the spelling achieve- 
ment of fifth-grade children. 

+ 


PROCEDURE 


Subjects: Three sections of fifth-grade Ss were selected 
for participation in the program.? In order to have a popu- 


1 Fifth-grade students were used as subjects because the 
experimental period was scheduled for the last four weeks of 
the academic year. Since the spelling materials were based on 
sixth grade spelling lists it was felt that these subjects would 
be more typically sixth graders (because it was the end of the 
year) therefore the material should be most appropriate for 


them. 
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lation representative of the school district? in which the 
experiment was conducted, one grade each was selected 
from three of the four elementary schools. The 96 Ss 
were then randomly assigned to one of six experimental 
conditions. To reduce the effects of the teacher variable, 
the students were assigned to one of six groups within 
each room, thereby creating six experimental conditions 
within each room. The mean I.O. of the group was 
105.58 and the mean age at the beginning of the experi- 
mental period was 11 years 1 month. As a further control, 
the students were matched on the basis of sex before 
assignment. 


Experimental Treatment: A list of 166 words was com- 
piled by selecting words most frequently included in 13 
widely used sixth-grade spellers. To compare the effects 
of size-of-step and cueing on learning, six different pro- 
grammed forms of each unit were written.3 The forms 
were: a small-step program (4-9 frames per word) in two 
forms, one with pictorial cues, and one without pictorial 
cues; a medium-step program (3-7 frames per word), one 
form with pictorial cues, the other form without pictorial 
cues; and a large-step program (3-6 frames per word) 
with the comparable forms as indicated for the first two 
programs. For pictorial cues, an illustration which was 
appropriate for each word was sketched on the program 
beside the word on its first occurrence. All materials were 
provided in a self-teaching textbook with a separate book- 
let containing the correct answers. The students were 
instructed to complete each item, then immediately con- 
firm the correctness of their response from the answer 
booklet. Other than general instructions for using the 
materials, no assistance in the teaching of spelling was 
provided. All Ss were instructed to complete each frame 
and proceed to the next one without reviewing previous 
ones, 

The total experimental period lasted for four weeks, 
during which Ss completed two units each week. The Ss 


*This study was conducted in the Danville area schools, 
Danville, Pennsylvania. The cooperation of Mr. Robert Hauck, 
elementary supervisor, and his staff made the study possible, 

The Programming was done by Mary Haupt Smith, 
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were given a maximum of 35 minutes for the completion 
of each unit. Tuesdays and Thursdays were used by all 
three teachers for completing the units. On Friday an 
orally administered spelling test consisting of a list of all 
words included in the two units was given by the teachers 
to the groups. This procedure was used on all four weeks 
of the experimental periods. 

In addition to the weekly tests, the Ss were pre-tested 
and post-tested respectively one day before and one day 
following the experimental period. This spelling test con- 
sisted of a random selection of 50 words out of the total 


spelling list. 


RESULTS 


To obtain evidence that the various experimental 
methods were effective in teaching spelling, the mean 
gain between the pre-test and post-test for all Ss was 
computed. The mean gain was 11.23 words. Since these 
tests were made up of a representative sample the spell- 
ing list of 166 words, an estimate of the total mean gain 
in spelling achievement can be obtained by multiplying 
the mean (11.23) of the sample by 3.33. The estimated 
mean gain in achievement over the four weeks of the 
experimental period was 40.96 words. 

To evaluate the effects of the several experimental con- 
ditions on the Ss’ spelling achievement, the means on 
each of the weekly tests were compared by the use of the 
F-test. 

In none of the four comparisons were the differences 
between means sufficiently great to reach a satisfactory 


level of significance. 


Discussion: Although only preliminary analysis of the 
data has been completed at this time, neither the size-of- 
step nor the pictorial cues seemed to affect significantly 
the scores on the criterion measure. This would suggest 


that the range in size-of-step and cueing frôm the small- 


+The 35-minute periods proved to be a sufficient amount of 
time for even the children who were the slowest workers in 
the groups. If a student completed the unit before the maximum 
time, he was instructed to work on other subjects. 
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step-pictorially-cued program to the large-step-non-pictori- 
ally-cued program all met the conditions necessary for 
learning. Since the large-step program proved to be as 
effective in teaching spelling as the small-step program 
(and it could be considered more efficient since logically 
it would require less time to complete) it may be con- 
cluded that programs consisting of steps which are very 
small are not necessarily an efficient method of instruc- 
tion. Further, if very small steps and over-cueing are 
interpreted by S as unnecessary repetition, then the addi- 
tional hazard of loss of interest in the material may be 
an undesirable by-product. Since this is a preliminary 
report, error rate on the individual programs has not been 
computed, therefore, comparison of error rate and 
achievement on the criterion measure have not been 
made. If it is found that a greater number of errors was 
made on the largest step, unillustrated program than on 
the smallest step, illustrated program, there will be sup- 
port for Goldbeck’s finding that a low error rate on the 
instructional materials does not always yield highest 
achievement on the criterion measure. 

It should be noted that all six forms of the program 
provided an effective method of teaching spelling without 
the assistance of a classroom teacher. 

It is concluded from these preliminary data that the 
necessity for small steps and strong cueing to assure the 
eliciting of the correct response to the frame of a pro- 
gram may have been overemphasized; that larger step 
programs (an optimal level must be identified) may have 
greater motivational value, and they may teach as much 
as effectively in less time. 
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Automated Teaching and Individual 
Differences* 


Jonn E. Courson AND Harry F. SILBERMAN 


System Development Corporation 


In 1958 an automated teaching project was formed by 
members of the Research Directorate at the System De- 
velopment Corporation. The functions of this project 
were to investigate the current status of automated teach- 
ing, to conduct research on variables of automated in- 
struction, and to determine what contribution SDC, as 
a non-profit corporation, might make in the develop- 
mental and applied aspects of this field. 

Project members began with a review of the “state of 
the art” as it then existed. This review indicated that, 
despite an apparent diversity in design, most teaching 
machines had one characteristic in common: a telative 
inflexibility of operation. With a given set of instruc- 
tional materials (“items”) in the machine, each student 
machine would receive the same se- 


quence of items, regardless of his individual capabilities 


or limitations. A 
The basic assumption underlying the design of a fixed- 


sequence teaching machine is that a single sequence of 
items can be optimally effective for a number of stu- 
dents. This assumption seemed highly doubtful when 


* This article originally appeared in Audio Visual Com- 
munication Review, 1961, 9, 5-15, and is reprinted here with 
the permission of the authors and the publisher. 
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viewed in the context of the typical classroom, with 20 
or more students differing not only in training experience 
but in inherent ability as well. It appeared that an ideal 
Sequence of items for one student would be less than 
maximally effective for some other student. 

Using certain behavior characteristics of the human 
tutor as an initial guide, project members considered 
possible teaching-machine designs which might use dif- 
ferent instructional procedures for different students, de- 
pending on the students’ performance during training. 
‘This adaptable feature was termed “machine Tesponsive- 
ness.” It was clearly recognized that well-written, well- 
tested instructional items were essential for any successful 
automated instruction. Without such materials any 
teaching machine would be completely useless. Given a 
good set of items, however, it was believed that the inter- 
action between student tesponse-behavior and machine 
selection of materials might be an important factor in the 
effectiveness of automated instruction. 

In April 1959, the Automated Teaching Project con- 
ducted an experiment with manually operated teaching 
machines to explore the effectiveness of machine re. 
sponsiveness, as compared with a fixed sequence mode 
(Coulson and Silberman, 1960). Responsiveness was in- 
troduced by a simplified form of “branching” in which 
a student who correctly answered an item on a given 
topic was not shown other items on the same topic.t In 
this way each member of the “branching” group received 
a (different) subset of the items seen by the fixed se- 
quence group. Other variables investigated in the same 
experiment were student response-mode (multiple-choice 
versus constructed-response) and item step-size (many 
small steps versus fewer large steps). Materials in elemen- 
tary psychology were presented to 80 junior-college stu- 
dents. Results of the study were as follows: 

1. Training with the manually controlled machines 
yielded significant student learning in each of the experi- 
mental groups. 


* This particular method of branching was unquestionably 
crude, and the criterion for branching somewhat unrealistic. It 
was found, however, that more complex forms of branching 
were not feasible with the manually operated machines, 
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2. “Branching” students required significantly less 
training time than “fixed sequence” students, and did 
not differ on a post-training criterion test. 

3. Students receiving many items with small steps 
learned more than students with fewer large-step items, 
but also required significantly greater training time. 

_ 4. “Multiple-choice” students required less training 
time than “constructed response” students; the two 
groups did not differ on criterion performance. 

The results of the initial experiment, while not defini- 
tive, pointed toward an advantage in favor of machine 
responsiveness over the fixed sequence mode of instruc- 
tion when training time was taken into consideration. 
At the same time, experience with the manual devices 
clearly indicated the desirability of more automatic equip- 
ment capable of controlling complex patterns of instruc- 
tional variables. Project members believed that a com- 
puter-based teaching machine would offer maximal re- 
sponsiveness to individual differences, as well as a high 
degree of flexibility as an experimental tool. 

Before designing a teaching machine with responsive- 
ness to individual differences, however, it was necessary 
to explore the effects of negative reinforcement in auto- 
mated teaching. 


Since a “responsive” teaching machine uses student 


errors to determine the sequence of items, it follows that 
most students will make at least a few errors during each 
training session. If this were not true, all students would 
receive the identical sequence. Consequently, an experi- 
ment was performed to determine the effects on learning 
of student errors and resulting negative reinforcement 
(Melaragno, 1960). Again manual equipment was used, 
this time to present instructional items in logic to 28 
junior-college students. Results indicated that a relatively 
small number of errors (negative reinforcements) dis- 
persed among correct responses did not affect student 
learning adversely, but that massed negative reinforce- 
ments did hinder learning. The conclusioti drawn was 
that student errors could be used for sequence control 
in a responsive teaching machine, but that items should 
be so written that the student would not miss many 


” items in a row. 
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COMPUTER-BASED TEACHING MACHINE 


tal teaching machine consisting of three major units: (1) 
a Bendix G-15 computer; (2) a random access slide pro- 
jector; and (3) an electric typewriter. 

l. Bendix G-15 Computer. This serves as the central 
control unit for the teaching machine. It contains the 
master control program which governs the operation of 

ine. The computer determines, at all 
times during a training session, what materials are to be 
presented to the student. It analyzes student responses to 
the Instructional material and compares these responses 


In early 1960, SDC completed work on an experimen- 


3. Electric Typewriter. Two primary functions are 
served by the typewriter, which is linked to the Bendix 


his answers to the instructional materials, normally in 
multiple-choice form. Second, the computer program 
takes control of the typewriter to print messages telling 
the student how successful he has been in answering the 
questions. 


FUNCTIONAL CHARACTERISTICS OF EXPERIMENTAL 
MACHINE 


The student begins with a “basic series” of items, each 


of which he attempts to answer in multiple-choice fash- k 
ion by pressing a key on the typewriter keyboard. He 
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receives appropriate knowledge of results after each at- 
tempt. Every item presented to him is registered within 
the computer control unit as having relevance to one or 
more specific topics to be taught. The computer keeps a 
record of the student’s performance on each item and a 
cumulative record of his performance on each of the 
topics covered by the items. These performance measures 
may include the student’s response times as well as his 
error count. If student performance falls below a certain 
level for a particular topic, the student is branched to a 
special set of items which provide remedial work on that 
topic. He may be taken through the entire remedial rou- 
tine, or may be brought more quickly back to the basic 
series if he performs well on the remedial items. 

On returning from the remedial routine the student 
again receives items from the basic series, continuing 
until he has finished this series or until his performance 
once more falls outside certain programmed limits. In the 
latter case, he is again branched temporarily from the 
basic series. 

In certain cases, a single missed item on a particular 
topic may be sufficient to take the student to a remedial 
routine. The teaching machine not only recognizes that 
an item has been answered incorrectly, but also considers 
the specific incorrect alternative chosen. Certain answers 
may be indicative of a serious misunderstanding by the 
student and may cause the machine to branch immedi- 
ately to a remedial sequence. 

If a student does very poorly in his training and re- 
quires excessive remedial work (according to programmed 
criteria), the machine may take him completely out of 
the original series, branching him into an alternative basic 
series. This alternative series represents not simply a 
remedial routine, but a major change in the training 
approach. The original series might, for instance, use the 
procedure of first giving general rules, and then present- 
ing examples to illustrate the rules; the alternative series 
might reverse the order, gradually building’up to a gen- 
eral rule by a progression of illustrative examples. 

In the event that a student’s performance on some 
topics of a basic series is sufficiently high, the teaching 
machine skips certain items for these topics, thereby 
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Fig. 1. Flow diagram showing types of decisions which the 
computer can be programmed to make during a lesson. 
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allowing the brighter student to progress rapidly to more 
advanced topics. 

At various points during the training session the stu- 
dent may be required by the machine to indicate his 
degree of confidence concerning his own learning prog- 
ress. If he expresses a feeling of confusion or lack of 
understanding, he may be taken to a remedial branch 
even though his actual performance is above the normal 
criterion level for such a branch. An expression of great 
self-confidence, on the other hand, may cause the student 
to be returned more rapidly from a remedial routine to 
the main sequence. 

The operation of the experimental teaching machine 
may be illustrated by means of the flow chart opposite 
(Fig. 1). This chart does not represent an actual instruc- 
tional sequence, but is merely used to demonstrate some 
of the types of decisions that can be made by a computer- 
based teaching machine. Each of the numbered squares 
symbolizes an instructional item. After the student has 
answered questions in items 1 and 2, the computer deter- 
mines whether he has missed either question. If he has, 
the computer gives him remedial items 3 and 4. If not, 
he goes directly to item 6 (representing a new topic). 

On item 4 the correct answer, b, takes the student 
directly to item 6. Answer a, representing a particular 
type of error by the student, causes him to receive item 3 
again. Answer c, another error, causes him to be given 
an extra remedial item, 5, before going ahead to item 6. 

Item 12 represents a self-evaluation item, in which the 
student is asked how well he feels he is doing. Answer a 
(eg., “I think I am doing well and would like to finish 
now”) leads him directly to the end of the lesson. An- 
swer b (“I’m doing fine, but I would like some review ) 
takes him to item 8 for a brief review of the materials. 
Answer ¢ (“I don’t understand this lesson at all”) takes 
him back to the start of the entire lesson. 


> 


EXPERIMENTS COMPLETED 


Following the development of the experimental teach- 
ing machine, research was started on a number of method 
variables, task variables, and student variables in auto- 
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mated teaching. The first study (Silberman et al., 1960) 
investigated the effects of one form of machine respon- 
siveness or branching, using over 400 items in logic. In 
this study the full capability of the automatic teaching 
machine was available. Thus a much more elaborate 
branching procedure was possible than that used in the 
previous studies with manual equipment. 

It was hypothesized that the performance of students 
receiving a branching sequence would be superior to that 
of students receiving a fixed sequence, with respect to 
Scores on a criterion post-test. In the branching group a 
student who had made errors on relatively difficult “main 
stream” items covering a topic was branched to a longer 
remedial list of simpler items covering the same topic. It 
was anticipated that by providing extra remedial items to 
students who had difficulty, the machine would minimize 
the number of items seen by the brighter student and the 
number of errors made by those with low aptitude. The 
over-all effect was expected to increase the amount of 
learning in a given time for each individual. 

Student aptitude and training time as well as instruc- 
tional materials were statistically equated for the two 
groups. With this design, obtained differences could not 
be attributed to differences in student scholastic aptitude, 
to amount of study time, nor to the different characteris- 
tics of the materials that were presented to the two 
groups, but only to the responsiveness of the machine to 
student errors when branching to remedial material. 

Mastery of the topics covered by the training items was 
tested immediately after the training session. Analysis of 
the test scores yielded no significant difference between 
the two groups. Several factors indicated that this finding 
resulted from the particular method of branching used 
and from the Se eg of the remedial items. Detailed 
analysis of item errors showed that the low aptitude stu- 
dents who received remedial materials did not benefit 
from them since they committed the same kinds of errors 
on the remed‘al materials as on main-stream items. The 
branching procedure also permitted only one trial 
through the items, so that even if the student still did 
not understand the topic after completing the remedial 
items he was sent on to the next topic. In addition, the 
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remedial items differed from the main-stream items only 
in the size of step or difficulty level, rather than in the 
approach used to cover the topic. 

Another experiment (Melaragno et al., 1960), this 
one using manual equipment, compared an optional 
branching procedure with the fixed sequence method. 
Whereas the previous study used branching which de- 
pended on student response errors, this study was 
designed to evaluate a procedure in which the student 
was allowed to branch at his own discretion. Three 
groups of high-school seniors were drawn from five 
schools. The first group received a set of teaching items 
in fixed sequence. Students in this group read each item 
in turn and were not permitted to view items already 
completed. Students in the second group received the 
same items as those in the first group but were permitted 
to back up one item at a time and review previous items 
that had already been covered. The third group received 
a five-page, single-spaced test consisting of the same ma- 
terials presented to the other two groups, except that 
teaching items were cast into statement form and organ- 
ized into paragraphs. Members of the third group were 
permitted to branch at their own option either to carlier 
material or ahead to material not yet covered. Mastery 
of the -topics was tested immediately after the training 
session. The analysis of test scores showed a significant 
difference between the optional branching group and the 
fixed sequence group in favor of branching. 


EXPERIMENTS PLANNED 


Several experiments are now being planned for 1961. 
The logic items which were written for the branching- 
fixed-sequence study are now being revised and the pro- 

ram structure extensively modified for another experi- 
mental attack on this problem. The branching decisions 
will be made not only on the basis of errors but also on 
the basis of student response time and self-evaluation. 
If a student takes too much time on certain topics or 
feels that he needs some review, extra remedial items 
will be presented to him. A much greater variety of ap- 
proaches will be used in the remedial items and students 
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will be halted until they have satisfactorily learned each 
topic. 

Ponoithies study being planned will not use the com- 
puter-controlled machine. One group of students will be 
required to give overt responses to instructional items 
presented manually. Students in a second group will re- 
ceive the same materials, but cast in the form of state- 
ments; members of this group will not be required to 
give an overt response nor permitted to branch among 
items. A third group will receive the same materials and 
conditions as the second group except that branching 
will be encouraged. Learning will be compared for the 
three conditions. This design is expected to provide some 
clarification of recent conflicting evidence concerning the 
relative effectiveness of these methods. 

SDC is planning a third project to develop an exten- 
sive sequence of items in high school mathematics using 
the computer-controlled teaching machine. These items 
will be used in an experiment in a number of high 
schools to answer certain practical questions concerning 
the application of auto-instructional methods in the pub- 
lic schools. The study will not be primarily concerned 
with the general question of whether auto-instructional 
methods teach, but with a determination of the kinds 
of students and the kinds of materials for which these 
methods are most effective. It will use two types of mate- 
rials: rote and conceptual. The rote material will require 
specific definitions and memorization of factual detail. 
The conceptual material will require a progression of 
thought leading to principles which generalize to novel 
problem situations. Three kinds of students will be used: 
over-achievers, normal-achievers, and under-achievers. 
The results of this study should yield valuable informa- 
tion on methods of applying different self-instructional 
materials to students of different motivational character- 
istics. It is expected that when conventional teaching 
methods are used, over-achievers will perform better than 
under-achievers on rote materials which have little in- 
trinsic appeal. On the more interesting conceptual topics, 
little difference is expected between the two groups. 
When self-instructional materials are used to supplement 


the regular instruction, it is expected that performance `. 
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for under-achievers and over-achievers will be essentially 
equated as the result of elimination of motivational dif- 
ferences between the two groups. This equating of per- 
formance is expected for both rote and conceptual mate- 
tials. The net effect of auto-instruction is expected to be 
greater for under-achievers than for over-achievers on 
tote materials, but no different for the two groups on 
conceptual material. 


FUTURE DEVELOPMENTS 


The computer-controlled teaching machine presently 
used at SDC can teach only one student at a time. Plans 
are now being developed for an expanded facility, using 
a larger computer control unit, that will give automated 
instruction to 20 students simultaneously. The same facil- 
ity will provide automatic storage, analysis, and printout 
of administrative and counseling data normally associated 
with an operational school system. Development of the 
facility will permit the study of important interactions 
among students, teachers and school administrators. 
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Teaching Machines and Individual 
Differences* 


ARTHUR R. JENSEN 
University of California 


In science it is not unusual for a new method or ap- 
paratus or technique to give rise to new areas of basic 
research, At times technical innovations will uncover a 
realm of phenomena never before accessible to scientific 
investigation. But, more often a technical development 
will merely force into greater prominence an already 
known but perhaps long neglected realm of phenomena. 
The scientific study of this realm then becomes a neces- 
sity if the technical innovation is to realize its full poten- 
tial in scientific research and ultimately in its application 
to practical problems. 

Such is the case with teaching machines. The recent 
upsurge of this innovation in educational technology 
forces us to confront a number of fundamental psycho- 
educational problems. Of course, the research that first 
comes to mind with respect to teaching machines is 
bound to have a practical and technical orientation, deal- 
ing with problems of design, programming, the compara- 
tive efficiency of teaching various subjects, and so forth. 
Not far behind these more immediate practical considera- 
tions will be“some concern with many of the facts, con- 


* This article originally appeared in Automated Teaching 
Bulletin, 1960, 1, 12-17, and is reprinted here with the per- 
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cepts, and theories derived from the experimental psy- 
chology of human learning. Thus, teaching machines are 
of interest not only to the applied psychologist, but also 
to the psychologist devoted to so-called “pure” or “basic” 
research on human learning. Most of the interest in 
teaching machines in academic circles today has been 
evinced either by experimental psychologists or by educa- 
tional psychologists who seem to be more interested in 
the immediate practical applications of teaching ma- 
chines. So far it appears that differential psychologists, 
ie., those concerned with the study of individual differ- 
ences, have not shown an active interest in teaching ma- 
chines. 

Perhaps because of certain historical accidents in the 
development of scientific psychology, such as the fact 
that experimental psychology very early took the physical 
sciences as its model, the experimental study of human 
learning and the study of individual differences have 
developed quite separately. These two fields—experimen- 
tal psychology and differential psychology—traditionally 
have left cach other pretty much alone. As a result, there 
is a large and rather clearly defined gap in our psychologi- 
cal knowledge of human behavior. This strange and 
bewildering gulf in psychology, which continues to per- 
sist, has been a detriment to our efforts to gain a greater 
understanding of human behavior. Now it appears that 
research on teaching machines may ultimately force a 
rapprochement between the experimental psychology of 
learning and the study of individual differences in learn- 
ing. 
The presence of individual differences in school per- 
formance are so obviously great as to have become tradi- 
tionally one of the primary concerns of educational 
psychology. But nearly all of the efforts to measure indi- 
vidual differences have been what might be called “static.” 
Rather than studying individual differences in the dy- 
namics of behavioral change, we have studied only the 
end products of learning by means of ov intelligence 
tests, aptitude tests, and achievement tests. How people 
differ in the processes by means of which behavioral 
change takes place is truly an unknown and unexplored 


territory in psychology. 
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Individual differences in learning will not be overcome 
or will not suddenly become unimportant when teaching 
machines are used. In all probability individual differ- 
ences will be accentuated. The teaching machine, far 
more effectively than the human teacher, can capitalize 
on individual differences if they are properly taken into 
account. I am not referring here only to such gross and 
obvious individual differences as time or number of trials 
needed to learn a given quantity of material. There are 
undoubtedly many othet more complex ways in which 
people differ in learning. For example, there are learning 
curves for two persons which are almost identical under 
conditions of distributed practice, but which diverge 
greatly from each other under conditions of massed prac- 
tice. Among more or less equally intelligent college stu- 
dents one can observe great individual differences in ma- 
chine learning under conditions of subject-pacing and of 
machine-pacing. Some students seem to learn more easily 
and quickly when they work at their own pace, while 
others do better when the pace is set by the machine. 
Our usual intelligence tests and aptitude tests will be of 
little value in discovering the kinds of individual differ- 
ences that will increase our understanding of performance 
on teaching machines. The kinds of learning difficulties 
that will show up in working with teaching machines will 
not lend themselves to diagnosis by means of the pres- 
ently available psychological tests. At present we do not 
even know what we should measure in order to get a bet- 
ter understanding of how and why various persons differ 
in their performance on teaching machines. It seems safe 
to say that individual differences in performance on 
teaching machines cannot be acounted for merely in 
terms of a simple unidimensional concept of “intelli- 
gence” or “learning ability.” Therefore, in order really 
to understand performance on teaching machines, so that 
the diagnosis of learning difficulties can be effective and 
so that in the use of teaching machines subject variables 
(i.e., individual differences) as well as independent vari- 
ables (i.e., those controlled by the machine) can be 
taken into account to achieve maximum efficiency, we 
must know a great deal about individual differences in 
learning. 
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The study of individual differences in learning is im- 
portant for still another reason. Without such study we 
cannot properly assess the relative importance of other 
parameters in learning efficiency, such as schedules of 
reinforcement, spaced vs. massed practice, stimulus and 
response similarity, whole vs. part learning, etc. For ex- 
ample, Underwood (1959) has expressed the opinion 
that the variable of spaced vs. massed practice is probably 
of little importance as a factor in the efficiency of verbal 
learning. The basis for this opinion is merely the average 
difference between spaced and massed practice for a large 
number of subjects. In the experiments on this topic we 
find that individual differences in learning under spaced 
and massed practice conditions are treated merely as 
“error variance,” and, interestingly enough, this variance, 
which is really the “between subjects” variance, is large 
in proportion to the variance due to experimental vari- 
ables. This being the case, Underwood’s conjecture is 
not at all convincing. It may well be that some subjects 
do better and others do worse under massed than under 
distributed practice, so that only on the average does this 
particular variable appear to be of slight importance. The 
same thing may be true for many other variables, the 
effects of which are simply cancelled out when we aver- 
age a large number of subjects. Obviously what we need 
is the investigation of what experimental psychologists 
have always regarded as a nuisance known as “error vari- 
ance.” All of this “error variance” is, of course, not error 
in the sense of inaccuracy of measurement. Most of it is 
“between subjects” variance, i.e, variance due to indi- 
vidual differences, and it warrants scientific investigation 
in its own right. : 

So far in the history of psychology attempts to dis- 
cover laws of learning have ignored the individual differ- 
ence variables. It now seems reasonable to believe that 
learning theory will have to acknowledge the existence 
of individual differences if much of the theory is to be 
applicable to individual cases. “Laws of Jearning will 
have to include statements about individual differences. 

Because of this gap in our psychological knowledge, to 
which recent developments such as the teaching machine 
have at last forced our attention, the writer has conceived 
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the need for a long-term program of systematic research 
on individual differences in human learning. This pro- 
gram is already underway in the writer’s laboratory. 

The frst requirements of such a project are of a tech- 
nological and methodological nature. First of all, a highly 
standardized experimental procedure for investigating 
learning variables and measuring performance is needed. 
For this purpose we are using a specially constructed 
apparatus which has many points in common with teach- 
ing machines. In some ways it is more limited and yet 
more flexible than a teaching machine need be. 

It is essentially three parts: (1) a stimulus display 
unit, consisting of a screen onto which various stimuli 
can be projected, (2) a response unit, consisting of a 
panel of buttons which can be varied in number and po- 
sition, and (3) a control unit, which consists of (a) a 
“reader” for the punched tape on which the entire ex- 
periment is programmed, (b) a bank of electrical coun- 
ters which tabulate the subject’s performance throughout 
the course of learning, and (c) “criterion counters” 
which stop the machine when the subject has attained 
any set criterion of learning. The rate of stimulus presen- 
tation can be either subject-paced or machine-paced. 

The apparatus permits trial and error selective learning 
(the subject is “reinforced” by a tone when he presses 
the “correct” button corresponding to a particular stimu- 
lus), paired-associate learning, serial learning, discrimi- 
nation learning, probability learning, multiple-choice de- 
cision making, and abstract reasoning or problem solving. 
The stimulus materials are placed inside the stimulus dis- 
play unit in the form of slides. The order of stimulus 
presentation and the schedule of reinforcement are pro- 
grammed on telegraphic tape. Thus, many subjects can 
be run through an experiment automatically with perfect 
uniformity of experimental procedure, an important con- 
sideration in obtaining reliable measures of individual 
differences in performance. The subject may respond ver- 
bally or by pressing buttons which may be labelled in 
various ways. The use of buttons makes it possible to 
cleanly define and delimit the subject’s response reper- 
oire. 


For quite some time it will be necessary to keep our 
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experiments on a very simple level in terms of the con- 
tent of the learning, since we wish to avoid as much as 
possible the effects of previous leaming which may show 
up in the form of associations, coding tricks, mnemonic 
devices, etc. For the same reason, we are using non- 
verbal stimuli. Students are learning to associate pressing 
certain buttons in response to such stimuli as squares, 
circles, and triangles colored red, blue, and yellow. Since 
at present we are interested only in simple associative 
learning, the use of such simple, easily discriminable 
stimuli obviates the need for the student to learn the 
stimuli themselves as would be the case if we used non- 
sense syllables. Eventually, of course, it will become de- 
sirable to use more complex stimuli and to introduce 
verbal materials. One advantage of the present set-up is 
that it permits the experimenter to perform a great va- 
riety of learning experiments all of which still involve 
essentially the same experimental parameters. He can 
change one variable at a time while all others are held 
constant. Thus a continuity and integration of findings 
from many different experiments will be possible. This is 
a most important feature as our research is envisaged as 
a very long-term program. 

What is the underlying 
research program? It may 


plan or “philosophy” of this 
be considered an attempt to 
discover the basic dimensions underlying individual dif- 
ferences in various learning phenomena, much as differ- 
ential psychologists have sought to discover the dimen- 
sions underlying cognitive abilities or personality charac- 
teristics. Since learning depends upon the nervous sys- 
tem, a product of biological evolution, it seems safe to 
assume that there is a limit to the number of ways in 
which people differ in the learning process. That is, there 
should be a limited number of fundamental dimensions 
in terms of which individual differences in various kinds 
of learning may be described. It seems most unlikely that 
there should be no “structure” to learning ability in the 
human species, since such structure is tœ be found in 
every other natural biological function. It is assumed that 
the capacity for learning is one such biological function. 

The investigator need not blindly begin his search for 
these dimensions. Learning theory, especially that deriv- 
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ing from Clark Hull, offers us some clues as to the possi- 
ble nature of the dimensions we might discover. For 
example, individual differences in performance on learn- 
ing tasks might be thought of as a joint effect of individ- 
ual variations in such Hullian variables as habit strength 
(ease of formation of associations), drive strength, in- 
hibition (a function of effortfulness of response, repeti- 
tion, and non-reinforcement), and so on. For example, 
differences in habit strength may show up in terms of 
sheer speed of learning when subjects are equated on 
other variables. Differences in inhibition may show up in 
rate of extinction, in differential rates of learning under 
massed and spaced practice, and in the magnitude of 
reminiscence effects. At present we have no idea how 
many basic dimensions we will ultimately find. One may 
hypothesize, for example, a dimension of susceptibility 
to “interference” or response competition which would 
be expected to show up in experiments on proactive and 
retroactive inhibition and associative interference. On 
the other hand, individual differences in these “interfer- 
ence” phenomena may be derived from more fundamen- 
tal dimensions, perhaps, for example, from an interaction 
between habit strength and drive. These are the kinds 
of problems we are setting out to study. 

Another problem on which sooner or later large-scale 
research will have to be launched concerns the relation 
between acquisition and retention. It may be that the 
dimensions we discover underlying individual differences 
in learning or acquisition will not be the same as those 
involved in the long-term retention of what has been 
learned. 

The specific plan of research consists first of determin- 
ing, within the confines of our simple experimental 
set-up, the extent or degree of individual differences exist- 
ing in various learning phenomena: simple selective 
learning, extinction, reminiscence, learning under differ- 
ent degrees of task complexity, reinforcement schedules, 
distributed vs. massed practice, whole vs. part learning, 
stimulus and response generalization, transfer, retention, 
“learning to learn,” and “interference” phenomena such 
as proactive and retroactive inhibition and associative 
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interference. It is necessary to determine the reliability 
of measurement of individual differences on these phe- 
nomena. This usually involves testing the same subjects 
at least twice on equivalent forms of the learning task. 
The “dimensions” within this limited set of phenomena 
would be, we hope, teased out by means of factor analysis 
or some other appropriate form of multivariate analysis. 

Next we would explore the generality of these dimen- 
sions in other forms of learning than simple selective 
trial and error learning, €g» paired associate learning 
and serial learning, still keeping the learning content rela- 
tively simple. We would hope that the dimensions thus 
discovered in our simple, rigorous, and highly controlled 
laboratory experiments could later be reliably measured 
in individuals and could be identified in theit perform- 
ance under the less simple and less rigorous conditions - 
approximating those found in the practical use of teach- 
ing machines. First in this wilderness we must seek the 
basic dimensions in the most simple and “purified” 
forms of various learning phenomena. What is discov- 
ered and proved in the Taboratory may later be demon- 
strated in more gross practical situations. 

Still quite far in the future in regard to the work 
planned for the writer's laboratory is a third stage of this 
research. It will consist of relating the dimensions of 
learning to the factors already discovered in the realm of 
intellectual or cognitive abilities and in the personality 


domain. 


certain pract 
of our over-all program as outline 
ferences in serial learning and in t 
position curve so far have receiyed the greater part of our 
efforts, Also, some interesting linkages between individ- 
ual differences in serial learning phenomena 1n the labo- 
ratory and spelling in school have been discovered which 
e ay be considered, at least 
in part, as a problem of serial learming- Linkages be 
tween other laboratory learning phenomena and other 
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forms of school learning are being sought, so that the 
work of our laboratory is not completely divorced from 
practical problems of learning. 

It is hoped that eventually in the not too distant fu- 
ture a truly scientific and practically fruitful rapproach- 
ment will be achieved between learning theory, the ex- 
perimental psychology of human learning, differential 
psychology, the psychology of school subjects, and the 
applied technology of teaéhing machines. 
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Programmed Self-Instruction: 
Possibilities and Limitations* 


Gene C. Fusco 


School Administration Branch, 
U.S. Office of Education 


POSSIBILITIES 


Proponents of programmed self-instruction maintain 
that this is not a new method of teaching; the basic 
procedure is found as far back as the Socratic dialogue. 
What is new, they say, is the development of a science 
and a technology based on a method employed by the 
great teachers of the past. Thus, some aspects of the art 
of teaching may become a part of the science of learning 
through programmed self-instruction. 

Theoretically, the mechanical tutors represent labor- 
saving devices for the teacher since the self-instructional 
techniques are tailored to the learning speed of the indi- 
vidual student. This method provides the student with 
as much practice as he needs, permits the rapid learner 
to cover the material quickly and the slower student to 
proceed at his own pace. As a consequence, the classroom 
teacher may be freed from the burdensome and time- 
consuming tasks of presenting material and taking pre- 
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cious class time to repeat material for students who didn’t 
get it the first time. 

Advocates of self-instructional techniques further 
maintain that this method may be adapted to home 
study and can save the teacher from devoting time to 
the correction of individual assignments. Self-correcting 
homework materials, it may be noted, have been pre- 
pared in high school English by Educational Testing 
Service (Diederich, 1960). Self-instructional materials 
may also be used by temporarily homebound students 
who may keep up with assignments while unable to at- 
tend regularly scheduled classes. In addition, some pro- 
pose that programmed materials may be used in small 
high schools with limited curriculum offerings to meet 
the needs of students who request courses not included 
in the school program. 

Students working with self-instructional materials will 
be enabled to work privately and independently and, as a 
consequence, recognize the fruits of their own labor. It 
is contended that this will result in a lessening of in- 
vidious comparisons of their achievements with that of 
their classmates. 

According to Simon Ramo, in the high school of the 
future there will still be human teachers and classroom 
sessions, but each student will spend a large proportion 
of each day interacting with computer-controlled teach- 
ing machines. The machines will tutor the student, 
schedule the curriculum to suit his individual abilities, 
and provide his human teachers with a detailed descrip- 
tion of his academic strengths and weaknesses. Ramo 
foresees the need for highly skilled teachers who can work 
closely with subject matter specialists and “teaching 
engineers” in modifying and revising the teaching de- 
vices and programs (Ramo, 1957). 


LIMITATIONS 


There are row dozens of different types of devices that 
could be classified as teaching machines, most of which 
Tepresent minor variations of two, or at best, three or 
four basic designs. At present, good problem materials 
to be used with teaching machines, and procedures and 
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principles for the preparation of such materials have not 
kept pace with the development of the devices them- 
selves. There is a lack of extensive empirical data on the 
effectiveness of this method of teaching. Despite some 
favorable findings of research in self-instructional tech- 
niques, there is little evidence on the factors that make 
that particular machine design or set of instructional 
items effective. There is much to be learned about the 
preparation of good items, and there are hardly any cri- 
teria to determine whether the items in a program are 
“Sood.” 

P nether limitation is that the selection of a particular 
teaching machine places severe limitations on the type 
of variables the researcher can investigate. Teaching ma- 
chines, for the most part, are single-purpose devices with 
limited adaptability. Thus, it is possible for the machine 
to dictate the direction of research, whereas, ideally, re- 
search considerations should direct the development of 
the machine. 

Another problem in reference to the inflexibility of 
programmed self-instructional methods is that a particu- 
lar program may not be optimally effective for all mem- 
bers of a typical class. Students, after all, differ in both 
experience and in inherent ability, or “intelligence.” An 
“ideal” sequence of items for one student may be less 
effective for some other student. : : 

Teaching machines present items in an essentially 
predetermined sequence, permit the student to respond, 
and give him immediate feedback. But does the fact that 
a student is shown the correct answer, or selects the cor- 
rect answer after several attempts, insure that he has 
actually acquired knowledge? The machine does not 
“know” whether the student has mastered the material 
the way a human instructor may know through observa- 
tion and questioning. <— 

Professor Skinner, for example, insists that a student 
through machine instruction will be enabled to proceed 
at his own level, advancing as rapidly as ue can. Under 
these conditions, he says, the conventional grading sys- 
tem will have to be overhauled since a grade will be 
useful only in showing how far a student has gone. Thus, 
a “C” might mean that he is halfway through a course. 


